Extended results
LibraryUseBench, every metric. Best value in each column in bold.
#ModelScore ± 95% CIPass rateSimplicityProblem $
1
Opus 5.5mini-SWE
66.9 ±7.1
93.0% ±1.7
78.3 ±3.4
$0.664 ±0.097
2
Opus 5mini-SWE
55.1 ±6.7
87.0% ±3.6
68.5 ±2.8
$1.314 ±0.184
3
DeepSeek V4.1 Flashmini-SWE
43.8 ±5.3
90.5% ±2.3
52.1 ±2.4
$0.190 ±0.025
4
GPT-5.6 Terramini-SWE
40.0 ±6.6
82.2% ±4.1
57.7 ±2.7
$0.239 ±0.020
5
Sonnet 5mini-SWE
39.9 ±6.8
80.1% ±4.1
56.0 ±3.0
$0.919 ±0.118
6
GLM 5.3 Flashmini-SWE
35.0 ±5.2
87.3% ±2.1
47.2 ±2.2
$0.192 ±0.032
7
DeepSeek V4 Flashmini-SWE
34.8 ±6.4
88.0% ±2.3
49.3 ±2.4
$0.286 ±0.039
8
GPT-5.6 Lunamini-SWE
30.1 ±5.8
79.0% ±4.8
46.5 ±2.5
$0.042 ±0.004
9
Muse Spark 1.3mini-SWE
29.4 ±5.6
92.0% ±1.7
35.8 ±2.6
$1.940 ±0.363
Score ± is the 95% confidence interval; the other ± are standard errors. Score, pass rate and simplicity are percentages; pass rate and simplicity average finished cells, costs average every cell that recorded one.