benchgap
Calibration

FrontierSWE v2 → PostTrainBench

PostTrainBench is estimated from FrontierSWE v2 with a offset logistic curve fitted on 10 models measured on both: y = 0.3314 + (0.4223 − 0.3314) / (1 + exp(−78.14·(x − 0.5322))), R² = 0.86, cross-validated error 2.4 pp. It is used for 13 estimates.

Estimated modelFrontierSWE v2PostTrainBenchSource
Claude Fable 5.269.9%42.2%estimated ± 2.4 pp, medium confidence
Claude Sonnet 5.561.9%42.2%estimated ± 2.4 pp, high confidence
Gemini 4 Hydrogen53.1%37.5%estimated ± 2.4 pp, high confidence
Claude Fable 547.0%33.2%estimated ± 2.4 pp, high confidence
Claude Haiku 5.543.8%33.1%estimated ± 2.4 pp, high confidence
Grok 4.729.5%33.1%estimated ± 2.4 pp, high confidence
Muse Spark 1.325.8%33.1%estimated ± 2.4 pp, high confidence
Grok 4.625.3%33.1%estimated ± 2.4 pp, high confidence
Gemini 3.7 Flash20.3%33.1%estimated ± 2.4 pp, high confidence
Gemini 3.8 Flash19.6%33.1%estimated ± 2.4 pp, high confidence
Qwen3.8 Max15.8%33.1%estimated ± 2.4 pp, medium confidence
Muse Spark 1.212.0%33.1%estimated ± 2.4 pp, medium confidence
Inkling4.1%33.1%estimated ± 2.4 pp, medium confidence

Fitted on

ModelFrontierSWE v2PostTrainBench
Claude Opus 552.0%35.7%
Claude Opus 5.562.3%43.8%
Gemini 4 Argon55.0%40.4%
Gemini 4 Carbon61.2%41.0%
GLM-5.330.2%34.5%
GLM-5.3-Flash18.1%36.8%
GPT-5.6 Sol32.2%31.0%
GPT-6 Astra65.5%41.9%
Inkling 2 Small25.1%31.9%
Kimi K325.9%31.5%