benchgap
Calibration

VulcanBench Frontier v4 → DeepSWE

DeepSWE is estimated from VulcanBench Frontier v4 with a Hill curve fitted on 15 models measured on both: y = 0.0000 + (0.7887 − 0.0000)·x^6.00 / (0.59814^6.00 + x^6.00), R² = 0.76, cross-validated error 6.5 pp. It is used for 1 estimate.

Estimated modelVulcanBench Frontier v4DeepSWESource
GPT-6.1 Terra90.0%72.6%estimated ± 6.5 pp, medium confidence

Fitted on

ModelVulcanBench Frontier v4DeepSWE
Claude Fable 5.191.8%67.4%
Claude Fable 5.293.9%74.3%
Claude Opus 5.591.1%74.2%
Claude Sonnet 5.592.0%71.0%
DeepSeek V4.1 Flash 101788.6%77.0%
GPT-5.6 Luna84.2%67.2%
GPT-5.6 Sol87.2%72.7%
GPT-5.6 Terra89.1%69.6%
GPT-6.1 Sol88.8%71.9%
GPT-6 Astra89.3%74.1%
GPT-6 Luna71.9%66.6%
GPT-6 Sol86.8%68.8%
Grok 4.793.2%71.0%
Muse Wonder Full88.4%79.0%
Beam 1.0 501B-A23B64.7%43.3%