Calibration
FrontierSWE v2 → Bug Hunt Bench
Bug Hunt Bench is estimated from FrontierSWE v2 with a offset logistic curve fitted on 19 models measured on both: y = 0.2525 + (0.5064 − 0.2525) / (1 + exp(−70.36·(x − 0.5463))), R² = 0.76, cross-validated error 7.3 pp. It is used for 4 estimates.
| Estimated model | FrontierSWE v2 | Bug Hunt Bench | Source |
|---|---|---|---|
| Claude Fable 5 | 47.0% | 25.4% | estimated ± 7.3 pp, medium confidence |
| Gemini 3.7 Flash | 20.3% | 25.3% | estimated ± 7.3 pp, medium confidence |
| Muse Spark 1.2 | 12.0% | 25.3% | estimated ± 7.3 pp, low confidence |
| Inkling | 4.1% | 25.3% | estimated ± 7.3 pp, low confidence |
Fitted on
| Model | FrontierSWE v2 | Bug Hunt Bench |
|---|---|---|
| Claude Fable 5.1 | 56.3% | 43.0% |
| Claude Fable 5.2 | 69.9% | 58.0% |
| Claude Haiku 5.5 | 43.8% | 21.5% |
| Claude Opus 5 | 52.0% | 27.0% |
| Claude Opus 5.5 | 62.3% | 41.7% |
| Claude Sonnet 5.5 | 61.9% | 51.3% |
| Gemini 3.8 Flash | 19.6% | 18.0% |
| Gemini 4 Argon | 55.0% | 40.7% |
| Gemini 4 Carbon | 61.2% | 57.3% |
| Gemini 4 Hydrogen | 53.1% | 32.7% |
| GLM-5.3 | 30.2% | 19.0% |
| GLM-5.3-Flash | 18.1% | 17.7% |
| GPT-5.6 Sol | 32.2% | 42.0% |
| GPT-6 Astra | 65.5% | 45.0% |
| Grok 4.6 | 25.3% | 27.0% |
| Grok 4.7 | 29.5% | 29.0% |
| Kimi K3 | 25.9% | 21.0% |
| Muse Spark 1.3 | 25.8% | 32.2% |
| Qwen3.8 Max | 15.8% | 25.7% |