Calibration
LiveCodeBench v6 → SWE-bench Pro
SWE-bench Pro is estimated from LiveCodeBench v6 with a offset logistic curve fitted on 15 models measured on both: y = 0.0000 + (0.6971 − 0.0000) / (1 + exp(−15.72·(x − 0.7773))), R² = 0.95, cross-validated error 4.7 pp. It is used for 11 estimates.
| Estimated model | LiveCodeBench v6 | SWE-bench Pro | Source |
|---|---|---|---|
| A.X K2 | 84.0% | 50.8% | estimated ± 4.7 pp, high confidence |
| Agents-A1-4B | 59.6% | 3.8% | estimated ± 4.7 pp, medium confidence |
| BTL-3 | 88.1% | 58.3% | estimated ± 4.7 pp, high confidence |
| Gemma 4 12B | 72.0% | 20.1% | estimated ± 4.7 pp, high confidence |
| Granite 4.2 3B | 69.7% | 15.4% | estimated ± 4.7 pp, high confidence |
| LFM2.5-2.6B | 59.4% | 3.7% | estimated ± 4.7 pp, medium confidence |
| LLaDA2.2-mini | 28.1% | 0.0% | estimated ± 4.7 pp, medium confidence |
| Mellum2-12B-A2.5B-Instruct | 37.2% | 0.1% | estimated ± 4.7 pp, medium confidence |
| Mellum2-12B-A2.5B-Thinking | 69.9% | 15.8% | estimated ± 4.7 pp, high confidence |
| MiniCPM5-1B | 33.5% | 0.1% | estimated ± 4.7 pp, medium confidence |
| ZAYA1-8B | 65.8% | 9.3% | estimated ± 4.7 pp, medium confidence |