Calibration
SWE-bench Pro → SWE-bench Verified
SWE-bench Verified is estimated from SWE-bench Pro with a Hill curve fitted on 44 models measured on both: y = 0.4294 + (1.1152 − 0.4294)·x^2.67 / (0.53491^2.67 + x^2.67), R² = 0.95, cross-validated error 2.5 pp. It is used for 17 estimates.
| Estimated model | SWE-bench Pro | SWE-bench Verified | Source |
|---|---|---|---|
| Atria Dawn Preview | 59.6% | 82.2% | estimated ± 2.5 pp, high confidence |
| Claude Fable 5.1 | 81.2% | 94.6% | estimated ± 2.5 pp, medium confidence |
| Gemini 3.5 Flash | 55.1% | 78.6% | estimated ± 2.5 pp, high confidence |
| Gemini 3.5 Flash-Lite | 54.2% | 77.8% | estimated ± 2.5 pp, high confidence |
| GPT-5.4 | 57.7% | 80.7% | estimated ± 2.5 pp, high confidence |
| GPT-5.6 Luna | 62.7% | 84.4% | estimated ± 2.5 pp, high confidence |
| GPT-5.6 Sol | 64.6% | 85.7% | estimated ± 2.5 pp, high confidence |
| GPT-5.6 Terra | 63.4% | 84.9% | estimated ± 2.5 pp, high confidence |
| Grok 4.5 | 64.7% | 85.8% | estimated ± 2.5 pp, high confidence |
| Laguna S 2.1 | 59.4% | 82.0% | estimated ± 2.5 pp, high confidence |
| Ling 3.0 Flash | 56.6% | 79.8% | estimated ± 2.5 pp, high confidence |
| MiMo-V2.5 | 56.1% | 79.4% | estimated ± 2.5 pp, high confidence |
| MiMo-V2.5-Pro | 57.2% | 80.3% | estimated ± 2.5 pp, high confidence |
| Muse Spark 1.1 | 61.5% | 83.5% | estimated ± 2.5 pp, high confidence |
| Sakana Fugu | 59.0% | 81.7% | estimated ± 2.5 pp, high confidence |
| Sakana Fugu-Ultra | 73.7% | 91.1% | estimated ± 2.5 pp, high confidence |
| Step 3.7 Flash | 56.3% | 79.6% | estimated ± 2.5 pp, high confidence |