benchgap
Calibration

FrontierSWE v2 → Bug Hunt Bench

Bug Hunt Bench is estimated from FrontierSWE v2 with a offset logistic curve fitted on 19 models measured on both: y = 0.2525 + (0.5064 − 0.2525) / (1 + exp(−70.36·(x − 0.5463))), R² = 0.76, cross-validated error 7.3 pp. It is used for 4 estimates.

Estimated modelFrontierSWE v2Bug Hunt BenchSource
Claude Fable 547.0%25.4%estimated ± 7.3 pp, medium confidence
Gemini 3.7 Flash20.3%25.3%estimated ± 7.3 pp, medium confidence
Muse Spark 1.212.0%25.3%estimated ± 7.3 pp, low confidence
Inkling4.1%25.3%estimated ± 7.3 pp, low confidence

Fitted on

ModelFrontierSWE v2Bug Hunt Bench
Claude Fable 5.156.3%43.0%
Claude Fable 5.269.9%58.0%
Claude Haiku 5.543.8%21.5%
Claude Opus 552.0%27.0%
Claude Opus 5.562.3%41.7%
Claude Sonnet 5.561.9%51.3%
Gemini 3.8 Flash19.6%18.0%
Gemini 4 Argon55.0%40.7%
Gemini 4 Carbon61.2%57.3%
Gemini 4 Hydrogen53.1%32.7%
GLM-5.330.2%19.0%
GLM-5.3-Flash18.1%17.7%
GPT-5.6 Sol32.2%42.0%
GPT-6 Astra65.5%45.0%
Grok 4.625.3%27.0%
Grok 4.729.5%29.0%
Kimi K325.9%21.0%
Muse Spark 1.325.8%32.2%
Qwen3.8 Max15.8%25.7%