benchgap
Calibration

DeepSWE → SciCode

SciCode is estimated from DeepSWE with a Michaelis–Menten + offset curve fitted on 5 models measured on both: y = 0.1843 + 2.0000·x / (2.70295 + x), R² = 0.64, cross-validated error 7.6 pp. It is used for 14 estimates.

Estimated modelDeepSWESciCodeSource
Qwen4 Max75.7%62.2%estimated ± 7.6 pp, low confidence
Hy474.9%61.8%estimated ± 7.6 pp, low confidence
Pareto 26.974.0%61.4%estimated ± 7.6 pp, low confidence
MAI Code 273.2%61.0%estimated ± 7.6 pp, low confidence
SWE-273.0%61.0%estimated ± 7.6 pp, low confidence
Particle-172.6%60.8%estimated ± 7.6 pp, low confidence
ERNIE 5.5 Flash Preview71.2%60.1%estimated ± 7.6 pp, low confidence
Pareto 26.10 Preview69.9%59.5%estimated ± 7.6 pp, low confidence
Hy4 flash64.6%57.0%estimated ± 7.6 pp, low confidence
LongCat-2.564.6%57.0%estimated ± 7.6 pp, low confidence
Ultramarine 1 Flash61.5%55.5%estimated ± 7.6 pp, low confidence
SWE-2 Flash60.1%54.8%estimated ± 7.6 pp, low confidence
MAI Code 2 Flash59.9%54.7%estimated ± 7.6 pp, low confidence
Mercury 346.3%47.7%estimated ± 7.6 pp, low confidence

Fitted on

ModelDeepSWESciCode
Beam 1.0 501B-A23B43.3%49.2%
Beam 1.0 501B-A23B FP843.0%48.3%
Beam 1.0 501B-A23B NVFP442.7%46.9%
Ripple-65B Preview44.3%39.4%
Step 5 Preview67.7%58.9%