Grok 4.6

39.7± 1.1 (95% CI)

mini-SWE · pass rate 84.5% · simplicity 54.1 · $2.63 per library

Runs

Each cell is one full run: the model designs the library, then 3 implementers solve every problem with it. Click a cell to open that run.

TaskRun 1Run 2Run 3Mean ± std

arazu 33.3 27.6 34.331.7 ± 3.6

clirs 40.3 48.0 47.945.4 ± 4.4

heretical 32.4 36.8 37.535.6 ± 2.7

npr 48.9 50.6 37.445.6 ± 7.2

plaid 36.3 41.3 42.540.0 ± 3.3

pyda 42.0 44.6 44.943.8 ± 1.6

canon 48.2 46.1 50.748.3 ± 2.3

roadkill 10.2 11.5 10.310.7 ± 0.7

rubberband 57.5 63.5 61.160.7 ± 3.0

weft 34.6 33.4 35.834.6 ± 1.2

sapi 37.3 32.2 38.936.1 ± 3.5

simu 35.9 34.2 40.036.7 ± 3.0

uglypie 50.0 48.9 54.051.0 ± 2.7

umami 47.4 46.0 46.046.5 ± 0.8

wapi 24.4 31.4 28.728.2 ± 3.5