Calibration
SWE-bench Pro → NL2Repo
NL2Repo is estimated from SWE-bench Pro with a linear curve fitted on 24 models measured on both: y = 1.1337·x + -0.2056, R² = 0.56, cross-validated error 6.4 pp. It is used for 4 estimates.
| Estimated model | SWE-bench Pro | NL2Repo | Source |
|---|---|---|---|
| Atria Dawn Preview | 59.6% | 47.0% | estimated ± 6.4 pp, medium confidence |
| Claude Opus 5.5 | 89.9% | 81.4% | estimated ± 6.4 pp, low confidence |
| Claude Sonnet 5.5 | 81.3% | 71.6% | estimated ± 6.4 pp, low confidence |
| Laguna S 2.1 | 59.4% | 46.8% | estimated ± 6.4 pp, medium confidence |