GLM 5.3

41.9± 1.1 (95% CI)

mini-SWE · pass rate 84.1% · simplicity 57.1 · $21.06 per library

Runs

Each cell is one full run: the model designs the library, then 3 implementers solve every problem with it. Click a cell to open that run.

TaskRun 1Run 2Run 3Mean ± std

arazu 34.1 29.6 28.930.9 ± 2.8

clirs 52.4 52.9 64.756.7 ± 7.0

heretical 39.1 33.0 36.136.1 ± 3.1

npr 46.8 49.0 50.348.7 ± 1.8

plaid 36.8 39.0 36.637.5 ± 1.3

pyda 46.7 47.3 43.345.7 ± 2.1

canon 47.0 50.9 50.349.4 ± 2.1

roadkill 10.8 12.9 13.712.4 ± 1.5

rubberband 60.5 69.7 64.665.0 ± 4.6

weft 33.0 33.0 29.831.9 ± 1.8

sapi 39.8 36.2 45.140.4 ± 4.5

simu 40.4 42.9 46.243.1 ± 2.9

uglypie 51.6 57.0 58.255.6 ± 3.5

umami 49.0 48.2 44.247.1 ± 2.6

wapi 31.7 25.4 24.627.2 ± 3.9