No design phase. Each agent gets a real production library and one instruction: use it, and write as little code as possible.
Scores run 0–100: pass rate² × simplicity, averaged over every problem, with problems weighted equally. Error bars and ± show the 95% confidence interval, clustered by task (how it is computed).
How are the confidence intervals computed?
A model's score is its mean over all 242 problems. Each problem counts once: its three runs are averaged first, and a run that did not finish scores 0.
The problems of one task all use the same production library, so a model that handles that library well tends to do well on all of them. Their scores are not independent, and treating them as if they were would make the intervals look far narrower than the data supports. The standard error is therefore clustered by task (15 clusters), following Miller (2024), "Adding Error Bars to Evals":
where G is the number of tasks and n the number of problems. The ± shown is 1.96 × SE. The interval says how much the score could move on other tasks like these, not only on reruns of the same ones.
Paired differences use the same clustering on each problem's difference between two models. Both models solve the same problems, so the difficulty they share cancels, and a difference can be clear even when the two leaderboard intervals overlap.