Calibration

SciCode → Vals SWE-bench

Name: benchgap: LLM benchmark scores, measured and estimated
Creator: benchgap
License: https://creativecommons.org/licenses/by-nc/4.0/

Vals SWE-bench is estimated from SciCode with a Hill curve fitted on 8 models measured on both: y = 0.0000 + (0.7966 − 0.0000)·x^6.00 / (0.31757^6.00 + x^6.00), R² = 0.39, cross-validated error 5.3 pp. It is used for 20 estimates.

Estimated model	SciCode	Vals SWE-bench	Source
Sakana Fugu	60.1%	78.0%	estimated ± 5.3 pp, low confidence
Step 5 Preview	58.9%	77.8%	estimated ± 5.3 pp, low confidence
Sakana Fugu-Ultra	58.7%	77.7%	estimated ± 5.3 pp, low confidence
Beam 1.0 501B-A23B	49.2%	74.3%	estimated ± 5.3 pp, low confidence
Beam 1.0 501B-A23B FP8	48.3%	73.7%	estimated ± 5.3 pp, low confidence
Beam 1.0 501B-A23B NVFP4	46.9%	72.7%	estimated ± 5.3 pp, low confidence
Sage-A1.5	44.3%	70.1%	estimated ± 5.3 pp, low confidence
Muse Glimmer 30B	43.6%	69.3%	estimated ± 5.3 pp, low confidence
Hy3 Preview	41.2%	65.9%	estimated ± 5.3 pp, low confidence
A.X K2	41.0%	65.5%	estimated ± 5.3 pp, low confidence
Ling 3.0 Flash FP8	40.4%	64.4%	estimated ± 5.3 pp, low confidence
Granite 4.2 30B	38.8%	61.2%	estimated ± 5.3 pp, low confidence
Mercury 2.5	38.0%	59.4%	estimated ± 5.3 pp, low confidence
K-EXAONE 2.0	37.4%	57.9%	estimated ± 5.3 pp, low confidence
Spark-X2.6-4B	36.8%	56.4%	estimated ± 5.3 pp, low confidence
Granite 4.2 8B	36.1%	54.4%	estimated ± 5.3 pp, low confidence
LFM2.5-24B-A2B	32.7%	43.4%	estimated ± 5.3 pp, low confidence
Nemotron 3 Nano Omni 30B A3B	32.0%	40.7%	estimated ± 5.3 pp, low confidence
Nemotron 3.5 Lightning 30B A3B NVFP4	31.4%	38.4%	estimated ± 5.3 pp, low confidence
Agents-A1-4B	29.6%	31.6%	estimated ± 5.3 pp, low confidence

Fitted on

Model	SciCode	Vals SWE-bench
Gemini 3.5 Flash	53.1%	78.8%
Grok 4.3	47.3%	71.4%
Inkling-Small	48.7%	82.2%
Kimi K2.6	52.2%	76.2%
Ling 3.0 Flash	41.2%	65.2%
Nemotron 3 Ultra	44.6%	69.0%
Qwen 3.6 Max (preview)	47.0%	72.8%
Qwen3.7 Max	53.5%	68.8%