Calibration
NL2Repo → SWE-bench Verified
SWE-bench Verified is estimated from NL2Repo with a Michaelis–Menten curve fitted on 15 models measured on both: y = 0.9719·x / (0.10189 + x), R² = 0.77, cross-validated error 2.4 pp. It is used for 14 estimates.
| Estimated model | NL2Repo | SWE-bench Verified | Source |
|---|---|---|---|
| DeepSeek V4.1 Flash | 65.4% | 84.1% | estimated ± 2.4 pp, medium confidence |
| GLM-5.1 | 42.7% | 78.5% | estimated ± 2.4 pp, high confidence |
| GLM-5.2 | 48.9% | 80.4% | estimated ± 2.4 pp, high confidence |
| GLM-5.3 | 58.0% | 82.7% | estimated ± 2.4 pp, high confidence |
| GLM-5.3-Flash | 56.3% | 82.3% | estimated ± 2.4 pp, high confidence |
| Hy4 preview | 58.9% | 82.9% | estimated ± 2.4 pp, high confidence |
| MiniMax M2.7 | 39.8% | 77.4% | estimated ± 2.4 pp, high confidence |
| Qwen 3.6 Max (preview) | 42.9% | 78.5% | estimated ± 2.4 pp, high confidence |
| Qwen3.8-27B | 42.3% | 78.3% | estimated ± 2.4 pp, high confidence |
| Qwen3.8-Flash-Next | 48.1% | 80.2% | estimated ± 2.4 pp, high confidence |
| Qwen3.8 Max | 55.9% | 82.2% | estimated ± 2.4 pp, high confidence |
| Qwen3.8-Omni-Flash | 48.9% | 80.4% | estimated ± 2.4 pp, high confidence |
| Seed 2.1 Pro | 47.0% | 79.9% | estimated ± 2.4 pp, high confidence |
| Seed 2.1 Turbo | 43.7% | 78.8% | estimated ± 2.4 pp, high confidence |