Calibration
LiveCodeBench v6 → SWE-bench Verified
SWE-bench Verified is estimated from LiveCodeBench v6 with a offset logistic curve fitted on 32 models measured on both: y = 0.5753 + (0.7352 − 0.5753) / (1 + exp(−112.90·(x − 0.7754))), R² = 0.41, cross-validated error 9.7 pp. It is used for 10 estimates.
| Estimated model | LiveCodeBench v6 | SWE-bench Verified | Source |
|---|---|---|---|
| BTL-3 | 88.1% | 73.5% | estimated ± 9.7 pp, low confidence |
| Mellum2-12B-A2.5B-Thinking | 69.9% | 57.5% | estimated ± 9.7 pp, low confidence |
| ZAYA1-8B | 65.8% | 57.5% | estimated ± 9.7 pp, low confidence |
| Qwen3 235B 2507 | 51.8% | 57.5% | estimated ± 9.7 pp, low confidence |
| Magistral Medium | 50.3% | 57.5% | estimated ± 9.7 pp, low confidence |
| Magistral Small | 47.4% | 57.5% | estimated ± 9.7 pp, low confidence |
| LLaDA2.2-mini | 28.1% | 57.5% | estimated ± 9.7 pp, low confidence |
| Mellum2-12B-A2.5B-Instruct | 37.2% | 57.5% | estimated ± 9.7 pp, low confidence |
| MiniCPM5-1B | 33.5% | 57.5% | estimated ± 9.7 pp, low confidence |
| Mistral Medium 3 | 30.3% | 57.5% | estimated ± 9.7 pp, low confidence |
Fitted on
| Model | LiveCodeBench v6 | SWE-bench Verified |
|---|---|---|
| BTL-4 | 66.1% | 78.4% |
| Claude Opus 4.5 | 84.8% | 80.9% |
| dots3-note Preview | 91.5% | 78.4% |
| Flint | 80.3% | 74.2% |
| GLM-4.7 | 84.9% | 73.8% |
| GLM-4.7-Flash | 64.0% | 59.2% |
| Granite 4.2 30B | 75.8% | 57.0% |
| Granite 4.2 8B | 73.2% | 47.7% |
| Kimi K2.6 | 89.6% | 80.2% |
| Kimi K2 | 53.7% | 65.8% |
| Kimi K2.5 | 85.0% | 76.8% |
| LFM3-1.7B | 73.5% | 43.4% |
| MAI-Thinking-1 | 87.7% | 73.5% |
| Mellum2.1-12B-A2.5B-Thinking | 82.0% | 47.0% |
| MiMo-V2-Flash | 80.6% | 73.4% |
| MiniCPM5-2B | 69.1% | 46.4% |
| Nemotron 3.5 Super VL 120B A12B | 88.2% | 71.1% |
| Nemotron 3.5 Super VL 120B A12B NVFP4 | 87.4% | 71.1% |
| Nemotron 3 Ultra | 89.0% | 71.9% |
| Qwen3.5-122B-A10B | 78.9% | 72.0% |
| Qwen3.5-27B | 80.7% | 72.4% |
| Qwen3.5-35B-A3B | 74.6% | 69.2% |
| Qwen3.5 397B | 83.6% | 76.2% |
| Qwen3.6-27B | 83.9% | 77.2% |
| Qwen3.6-35B-A3B | 80.4% | 73.4% |
| Qwen3.6 Plus | 87.1% | 78.8% |
| Qwen3.7 Max | 91.6% | 80.4% |
| Qwen3.7 Plus | 89.6% | 77.7% |
| Solar Open 2 | 92.4% | 70.4% |
| Step 3.5 Flash | 86.4% | 74.4% |
| Ternary Bonsai 2 27B | 90.1% | 60.8% |
| ZAYA1-74B-Preview | 65.7% | 53.2% |