Extended results
LibraryUseBench, every metric. Best model in each column in bold.
#ModelScore ± 95% CIPass rateSimplicityProblem $
1
Opus 5.5mini-SWE
66.9 ±7.1
93.0% ±1.7
78.3 ±3.4
$0.664 ±0.097
2
Sonnet 5.5mini-SWE
63.8 ±7.0
91.8% ±1.8
75.7 ±3.3
$0.374 ±0.072
3
GPT-6 Solmini-SWE
57.6 ±6.0
92.3% ±2.0
66.8 ±2.6
$0.237 ±0.042
4
Opus 5mini-SWE
55.1 ±6.7
87.0% ±3.6
68.5 ±2.8
$1.314 ±0.184
5
DeepSeek V4.1 Flashmini-SWE
43.8 ±5.3
90.5% ±2.3
52.1 ±2.4
$0.190 ±0.025
6
GPT-5.6 Terramini-SWE
40.0 ±6.6
82.2% ±4.1
57.7 ±2.7
$0.239 ±0.020
7
Sonnet 5mini-SWE
39.9 ±6.8
80.1% ±4.1
56.0 ±3.0
$0.919 ±0.118
8
GLM 5.3 Flashmini-SWE
35.0 ±5.2
87.3% ±2.1
47.2 ±2.2
$0.192 ±0.032
9
DeepSeek V4 Flashmini-SWE
34.8 ±6.4
88.0% ±2.3
49.3 ±2.4
$0.286 ±0.039
10
GPT-6 Lunamini-SWE
33.5 ±7.5
71.4% ±5.5
62.8 ±3.8
$0.007 ±0.001
11
GPT-5.6 Lunamini-SWE
30.1 ±5.8
79.0% ±4.8
46.5 ±2.5
$0.042 ±0.004
Score ± is the 95% confidence interval; the other ± are standard errors. Score, pass rate and simplicity are percentages; pass rate and simplicity average finished cells, costs average every cell that recorded one.