Calibration
LiveCodeBench v6 → NL2Repo
NL2Repo is estimated from LiveCodeBench v6 with a Michaelis–Menten curve fitted on 5 models measured on both: y = 2.0000·x / (2.97945 + x), R² = 0.37, cross-validated error 3.1 pp. It is used for 26 estimates.
| Estimated model | LiveCodeBench v6 | NL2Repo | Source |
|---|---|---|---|
| A.X K2 | 84.0% | 44.0% | estimated ± 3.1 pp, low confidence |
| Agents-A1-4B | 59.6% | 33.3% | estimated ± 3.1 pp, low confidence |
| BTL-3 | 88.1% | 45.6% | estimated ± 3.1 pp, low confidence |
| BTL-4 | 66.1% | 36.3% | estimated ± 3.1 pp, low confidence |
| Gemma 4 12B | 72.0% | 38.9% | estimated ± 3.1 pp, low confidence |
| Granite 4.2 30B | 75.8% | 40.5% | estimated ± 3.1 pp, low confidence |
| Granite 4.2 3B | 69.7% | 37.9% | estimated ± 3.1 pp, low confidence |
| Granite 4.2 8B | 73.2% | 39.5% | estimated ± 3.1 pp, low confidence |
| Kimi K2.6 | 89.6% | 46.2% | estimated ± 3.1 pp, low confidence |
| Kimi K2.5 | 85.0% | 44.4% | estimated ± 3.1 pp, low confidence |
| LFM2.5-2.6B | 59.4% | 33.2% | estimated ± 3.1 pp, low confidence |
| LLaDA2.2-mini | 28.1% | 17.3% | estimated ± 3.1 pp, low confidence |
| MAI-Thinking-1 | 87.7% | 45.5% | estimated ± 3.1 pp, low confidence |
| Mellum2-12B-A2.5B-Instruct | 37.2% | 22.2% | estimated ± 3.1 pp, low confidence |
| Mellum2-12B-A2.5B-Thinking | 69.9% | 38.0% | estimated ± 3.1 pp, low confidence |
| MiniCPM5-1B | 33.5% | 20.2% | estimated ± 3.1 pp, low confidence |
| MiniCPM5-2B | 69.1% | 37.7% | estimated ± 3.1 pp, low confidence |
| Nemotron 3 Ultra | 89.0% | 46.0% | estimated ± 3.1 pp, low confidence |
| Qwen3.5 397B | 83.6% | 43.8% | estimated ± 3.1 pp, low confidence |
| Qwen3.6 Plus | 87.1% | 45.2% | estimated ± 3.1 pp, low confidence |
| Sakana Fugu | 92.9% | 47.5% | estimated ± 3.1 pp, low confidence |
| Sakana Fugu-Ultra | 93.2% | 47.7% | estimated ± 3.1 pp, low confidence |
| Solar Open 2 | 92.4% | 47.3% | estimated ± 3.1 pp, low confidence |
| Ternary Bonsai 2 27B | 90.1% | 46.4% | estimated ± 3.1 pp, low confidence |
| ZAYA1-74B-Preview | 65.7% | 36.1% | estimated ± 3.1 pp, low confidence |
| ZAYA1-8B | 65.8% | 36.2% | estimated ± 3.1 pp, low confidence |