GPT-6 Astra

45.1± 0.7 (95% CI)

mini-SWE · pass rate 85.7% · simplicity 58.7 · $3.63 per library

Runs

Each cell is one full run: the model designs the library, then 3 implementers solve every problem with it. Click a cell to open that run.

TaskRun 1Run 2Run 3Mean ± std

arazu 45.6 46.5 42.044.7 ± 2.4

clirs 59.2 58.4 55.557.7 ± 1.9

heretical 36.8 41.5 38.238.8 ± 2.4

npr 55.8 49.8 52.952.8 ± 3.0

plaid 44.5 43.7 48.145.4 ± 2.4

pyda 47.5 52.8 47.349.2 ± 3.1

canon 57.8 53.2 58.956.6 ± 3.0

roadkill 8.8 10.5 11.710.3 ± 1.5

rubberband 60.1 64.2 58.160.8 ± 3.1

weft 35.9 37.1 37.736.9 ± 0.9

sapi 38.3 41.7 39.139.7 ± 1.8

simu 43.3 44.2 43.043.5 ± 0.7

uglypie 60.9 62.6 58.960.8 ± 1.9

umami 46.3 48.2 46.146.9 ± 1.2

wapi 35.9 29.1 30.631.9 ± 3.6