P

sapi · Production library run 2

48.1Library score
Pass rate96.8%
Simplicity51.5
Problems $$2.44

Implementers use the real production library (fastapi).

ImplementerPass rateSimplicityCostScore
DeepSeek V4.1 Flash · mini-SWE97.2%54.9$0.58
51.8
GLM 5.3 Flash · mini-SWE96.9%53.1$1.06
49.2
GPT-5.6 Luna · Codex96.4%46.5$0.81
43.4
ProblemDeepSeek V4.1 FlashGLM 5.3 FlashGPT-5.6 LunaMean
fleet-api30.845.932.236.3telemetry-api64.255.848.656.2catalog-search44.655.440.746.9delivery-profile59.161.659.760.1shipment-preview46.146.230.941.1protected-notes38.531.227.632.5sensor-export43.845.246.645.2versioned-orders69.369.951.263.5browser-sessions39.034.935.436.4signed-webhook64.626.149.646.8service-catalog69.568.854.764.3