Calibration
LiveCodeBench v6 → SWE-bench Verified
SWE-bench Verified is estimated from LiveCodeBench v6 with a Michaelis–Menten curve fitted on 15 models measured on both: y = 2.0000·x / (1.55881 + x), R² = 0.32, cross-validated error 10.5 pp. It is used for 6 estimates.
| Estimated model | LiveCodeBench v6 | SWE-bench Verified | Source |
|---|---|---|---|
| BTL-3 | 88.1% | 72.2% | estimated ± 10.5 pp, low confidence |
| LLaDA2.2-mini | 28.1% | 30.6% | estimated ± 10.5 pp, low confidence |
| Mellum2-12B-A2.5B-Instruct | 37.2% | 38.5% | estimated ± 10.5 pp, low confidence |
| Mellum2-12B-A2.5B-Thinking | 69.9% | 61.9% | estimated ± 10.5 pp, low confidence |
| MiniCPM5-1B | 33.5% | 35.4% | estimated ± 10.5 pp, low confidence |
| ZAYA1-8B | 65.8% | 59.4% | estimated ± 10.5 pp, low confidence |