GPT-6 Sol

38.5± 0.7 (95% CI)

mini-SWE · pass rate 85.2% · simplicity 51.6 · $0.30 per library

Runs

Each cell is one full run: the model designs the library, then 3 implementers solve every problem with it. Click a cell to open that run.

TaskRun 1Run 2Run 3Mean ± std

arazu 34.4 37.8 36.436.2 ± 1.7

clirs 45.6 45.2 44.345.0 ± 0.7

heretical 31.9 27.5 24.928.1 ± 3.5

npr 38.5 41.5 45.842.0 ± 3.7

plaid 41.8 42.7 43.042.5 ± 0.6

pyda 43.8 44.4 45.744.6 ± 1.0

canon 50.0 53.0 47.550.2 ± 2.7

roadkill 8.9 9.5 7.08.5 ± 1.3

rubberband 50.0 51.2 46.249.1 ± 2.6

weft 29.8 29.4 28.229.1 ± 0.8

sapi 38.1 33.6 40.737.5 ± 3.6

simu 41.2 38.5 39.939.9 ± 1.3

uglypie 49.9 52.9 46.849.8 ± 3.1

umami 48.6 51.3 47.649.2 ± 1.9

wapi 26.6 23.2 28.025.9 ± 2.5