benchgap
Calibration

SWE-bench Pro → DeepSWE

DeepSWE is estimated from SWE-bench Pro with a Michaelis–Menten + offset curve fitted on 25 models measured on both: y = 0.0000 + 2.0000·x / (1.68402 + x), R² = 0.30, cross-validated error 11.2 pp. It is used for 10 estimates.

Estimated modelSWE-bench ProDeepSWESource
Claude Mythos 580.3%64.6%estimated ± 11.2 pp, low confidence
Atria Dawn Preview59.6%52.3%estimated ± 11.2 pp, low confidence
Claude Opus 4.653.4%48.2%estimated ± 11.2 pp, low confidence
MAI-Thinking-152.8%47.7%estimated ± 11.2 pp, low confidence
Laguna M.149.2%45.2%estimated ± 11.2 pp, low confidence
Laguna XS 2.147.6%44.1%estimated ± 11.2 pp, low confidence
Laguna XS.246.3%43.1%estimated ± 11.2 pp, low confidence
LongCat-Flash-Lite-Sparse40.6%38.9%estimated ± 11.2 pp, low confidence
LLaDA2.2-flash30.1%30.3%estimated ± 11.2 pp, low confidence
Mellum2.1-12B-A2.5B-Thinking28.0%28.5%estimated ± 11.2 pp, low confidence

Fitted on

ModelSWE-bench ProDeepSWE
Claude Fable 5.181.2%67.4%
Claude Fable 5.293.0%74.3%
Claude Opus 579.2%68.8%
Claude Opus 5.589.9%74.2%
Claude Sonnet 5.581.3%71.0%
DeepSeek V4 Flash 073152.6%54.4%
DeepSeek V4 Pro55.4%62.7%
GPT-5.6 Luna62.7%67.2%
GPT-5.6 Sol64.6%72.7%
GPT-5.6 Terra63.4%69.6%
Grok 4.564.7%53.0%
Hy4 preview65.7%64.3%
Laguna S 2.159.4%40.4%
Ling 3.1 Flash68.7%59.7%
Muse Spark 1.161.5%53.3%
Ornith-1.5-35B-A3B59.6%22.0%
Ornith-1.5-397B65.1%56.0%
Qwen3.8-27B61.7%42.2%
Qwen3.8-Flash-Next62.5%58.7%
Qwen3.8 Max67.7%56.6%
Qwen3.8-Omni-Flash63.3%57.8%
Beam 1.0 501B-A23B66.1%43.3%
Beam 1.0 501B-A23B FP865.9%43.0%
Beam 1.0 501B-A23B NVFP465.6%42.7%
Ripple-65B Preview61.4%44.3%