No design phase. Each agent gets a real production library and one instruction: use it, and write as little code as possible.
- Score ± 95% CI
- 66.9 ± 7.1
- Pass rate
- 93.0%
- Simplicity
- 78.3
- $ / problem
- $0.664
- Score ± 95% CI
- 63.8 ± 7.0
- Pass rate
- 91.8%
- Simplicity
- 75.7
- $ / problem
- $0.374
- Score ± 95% CI
- 57.6 ± 6.0
- Pass rate
- 92.3%
- Simplicity
- 66.8
- $ / problem
- $0.237
- Score ± 95% CI
- 55.1 ± 6.7
- Pass rate
- 87.0%
- Simplicity
- 68.5
- $ / problem
- $1.314
- Score ± 95% CI
- 43.8 ± 5.3
- Pass rate
- 90.5%
- Simplicity
- 52.1
- $ / problem
- $0.190
- Score ± 95% CI
- 40.0 ± 6.6
- Pass rate
- 82.2%
- Simplicity
- 57.7
- $ / problem
- $0.239
- Score ± 95% CI
- 39.9 ± 6.8
- Pass rate
- 80.1%
- Simplicity
- 56.0
- $ / problem
- $0.919
- Score ± 95% CI
- 35.0 ± 5.2
- Pass rate
- 87.3%
- Simplicity
- 47.2
- $ / problem
- $0.192
- Score ± 95% CI
- 34.8 ± 6.4
- Pass rate
- 88.0%
- Simplicity
- 49.3
- $ / problem
- $0.286
- Score ± 95% CI
- 33.5 ± 7.5
- Pass rate
- 71.4%
- Simplicity
- 62.8
- $ / problem
- $0.007
- Score ± 95% CI
- 30.1 ± 5.8
- Pass rate
- 79.0%
- Simplicity
- 46.5
- $ / problem
- $0.042
Line: best score for the cost. Hover or tap a point for details.
Dark edge: on the best-for-cost line.
Scores run 0–100: pass rate² × simplicity, averaged over every problem, with problems weighted equally.
How are the confidence intervals computed?
A model's score is its mean over all 242 problems. Each problem counts once: its three runs are averaged first, and a run that did not finish scores 0.
The problems of one task all use the same production library, so a model that handles that library well tends to do well on all of them. Their scores are not independent, and treating them as if they were would make the intervals look far narrower than the data supports. The standard error is therefore clustered by task (15 clusters), following Miller (2024), "Adding Error Bars to Evals":
where G is the number of tasks and n the number of problems. The ± shown is 1.96 × SE. The interval says how much the score could move on other tasks like these, not only on reruns of the same ones.
Paired differences use the same clustering on each problem's difference between two models. Both models solve the same problems, so the difficulty they share cancels, and a difference can be clear even when the two leaderboard intervals overlap.