DeepSeek V4 Pro

31.2± 0.8 (95% CI)

mini-SWE · pass rate 84.9% · simplicity 42.0 · $0.31 per library

Runs

Each cell is one full run: the model designs the library, then 3 implementers solve every problem with it. Click a cell to open that run.

TaskRun 1Run 2Run 3Mean ± std

arazu 19.1 18.9 20.219.4 ± 0.7

clirs 30.2 33.1 29.130.8 ± 2.1

heretical 33.7 24.9 28.529.0 ± 4.5

npr 34.7 37.9 34.335.6 ± 2.0

plaid 31.4 37.7 30.833.3 ± 3.8

pyda 37.3 34.2 35.335.6 ± 1.6

canon 36.1 32.0 40.236.1 ± 4.1

roadkill 5.6 8.2 7.16.9 ± 1.3

rubberband 39.0 32.7 40.937.5 ± 4.3

weft 29.8 31.3 30.330.5 ± 0.8

sapi 19.8 23.8 21.021.5 ± 2.1

simu 37.0 34.3 35.835.7 ± 1.4

uglypie 40.8 38.7 45.241.5 ± 3.3

umami 45.2 45.7 44.545.1 ± 0.6

wapi 31.9 28.9 28.129.7 ± 2.0