–

canon · No library run 2

35.2Library score
Pass rate93.9%
Simplicity38.5
Problems $$2.01

Implementers get no library and write everything from scratch.

ImplementerPass rateSimplicityCostScore
DeepSeek V4.1 Flash · mini-SWE99.3%38.6$0.62
38.1
GLM 5.3 Flash · mini-SWE92.0%39.7$0.74
34.9
GPT-5.6 Luna · Codex90.5%37.3$0.66
32.5
ProblemDeepSeek V4.1 FlashGLM 5.3 FlashGPT-5.6 LunaMean
profile-patch35.539.310.028.3thermostat-codec39.732.543.038.4tagged-config45.344.039.242.8rtl-pin-lock35.835.632.134.5pipeline-provenance22.74.223.716.9boundary-views27.714.527.023.0ml-run-config63.656.245.255.0ha-triggers50.444.947.647.6env-config43.849.048.247.0paper-poster52.143.358.751.3typed-client26.334.56.222.3cli-dispatch27.423.323.724.8trading-events38.736.225.633.5llm-contracts24.431.424.926.9