–

canon · No library run 1

34.1Library score
Pass rate93.7%
Simplicity37.6
Problems $$2.06

Implementers get no library and write everything from scratch.

ImplementerPass rateSimplicityCostScore
DeepSeek V4.1 Flash · mini-SWE97.7%37.2$0.62
35.8
GLM 5.3 Flash · mini-SWE94.9%39.7$0.76
35.7
GPT-5.6 Luna · Codex88.4%35.9$0.68
30.6
ProblemDeepSeek V4.1 FlashGLM 5.3 FlashGPT-5.6 LunaMean
profile-patch29.046.911.229.0thermostat-codec40.138.337.938.8tagged-config46.526.342.738.5rtl-pin-lock28.343.70.424.1pipeline-provenance21.520.822.621.6boundary-views29.126.725.927.2ml-run-config64.751.141.052.3ha-triggers49.251.649.250.0env-config41.136.041.739.6paper-poster53.841.655.050.1typed-client21.036.822.026.6cli-dispatch26.122.223.223.8trading-events30.732.031.931.5llm-contracts20.326.424.223.6