–

sapi · No library run 3

20.4Library score
Pass rate96.1%
Simplicity22.3
Problems $$1.41

Implementers get no library and write everything from scratch.

ImplementerPass rateSimplicityCostScore
DeepSeek V4.1 Flash · mini-SWE98.4%22.1$0.36
21.2
GLM 5.3 Flash · mini-SWE93.1%23.1$0.54
19.9
GPT-5.6 Luna · Codex96.8%21.6$0.51
20.1
ProblemDeepSeek V4.1 FlashGLM 5.3 FlashGPT-5.6 LunaMean
fleet-api21.116.421.219.6telemetry-api26.738.626.330.5catalog-search18.514.015.015.8delivery-profile14.011.913.713.2shipment-preview11.810.614.812.4protected-notes24.722.223.723.5sensor-export25.018.121.421.5versioned-orders20.422.118.420.3browser-sessions22.815.119.119.0signed-webhook32.535.532.533.5service-catalog15.914.215.515.2