benchgap
Calibration

LiveCodeBench v6 → SWE-bench Verified

SWE-bench Verified is estimated from LiveCodeBench v6 with a offset logistic curve fitted on 32 models measured on both: y = 0.5753 + (0.7352 − 0.5753) / (1 + exp(−112.90·(x − 0.7754))), R² = 0.41, cross-validated error 9.7 pp. It is used for 10 estimates.

Estimated modelLiveCodeBench v6SWE-bench VerifiedSource
BTL-388.1%73.5%estimated ± 9.7 pp, low confidence
Mellum2-12B-A2.5B-Thinking69.9%57.5%estimated ± 9.7 pp, low confidence
ZAYA1-8B65.8%57.5%estimated ± 9.7 pp, low confidence
Qwen3 235B 250751.8%57.5%estimated ± 9.7 pp, low confidence
Magistral Medium50.3%57.5%estimated ± 9.7 pp, low confidence
Magistral Small47.4%57.5%estimated ± 9.7 pp, low confidence
LLaDA2.2-mini28.1%57.5%estimated ± 9.7 pp, low confidence
Mellum2-12B-A2.5B-Instruct37.2%57.5%estimated ± 9.7 pp, low confidence
MiniCPM5-1B33.5%57.5%estimated ± 9.7 pp, low confidence
Mistral Medium 330.3%57.5%estimated ± 9.7 pp, low confidence

Fitted on

ModelLiveCodeBench v6SWE-bench Verified
BTL-466.1%78.4%
Claude Opus 4.584.8%80.9%
dots3-note Preview91.5%78.4%
Flint80.3%74.2%
GLM-4.784.9%73.8%
GLM-4.7-Flash64.0%59.2%
Granite 4.2 30B75.8%57.0%
Granite 4.2 8B73.2%47.7%
Kimi K2.689.6%80.2%
Kimi K253.7%65.8%
Kimi K2.585.0%76.8%
LFM3-1.7B73.5%43.4%
MAI-Thinking-187.7%73.5%
Mellum2.1-12B-A2.5B-Thinking82.0%47.0%
MiMo-V2-Flash80.6%73.4%
MiniCPM5-2B69.1%46.4%
Nemotron 3.5 Super VL 120B A12B88.2%71.1%
Nemotron 3.5 Super VL 120B A12B NVFP487.4%71.1%
Nemotron 3 Ultra89.0%71.9%
Qwen3.5-122B-A10B78.9%72.0%
Qwen3.5-27B80.7%72.4%
Qwen3.5-35B-A3B74.6%69.2%
Qwen3.5 397B83.6%76.2%
Qwen3.6-27B83.9%77.2%
Qwen3.6-35B-A3B80.4%73.4%
Qwen3.6 Plus87.1%78.8%
Qwen3.7 Max91.6%80.4%
Qwen3.7 Plus89.6%77.7%
Solar Open 292.4%70.4%
Step 3.5 Flash86.4%74.4%
Ternary Bonsai 2 27B90.1%60.8%
ZAYA1-74B-Preview65.7%53.2%