Calibration
NL2Repo → SWE-bench Pro
SWE-bench Pro is estimated from NL2Repo with a Hill curve fitted on 24 models measured on both: y = 0.3855 + (0.6329 − 0.3855)·x^6.00 / (0.33611^6.00 + x^6.00), R² = 0.72, cross-validated error 3.6 pp. It is used for 5 estimates.
| Estimated model | NL2Repo | SWE-bench Pro | Source |
|---|---|---|---|
| DeepSeek V4.1 Flash | 65.4% | 62.8% | estimated ± 3.6 pp, medium confidence |
| GLM-5.3 | 58.0% | 62.4% | estimated ± 3.6 pp, high confidence |
| GLM-5.3-Flash | 56.3% | 62.2% | estimated ± 3.6 pp, high confidence |
| Seed 2.1 Pro | 47.0% | 60.4% | estimated ± 3.6 pp, high confidence |
| Seed 2.1 Turbo | 43.7% | 59.0% | estimated ± 3.6 pp, high confidence |