Calibration
SWE-bench Pro → SWE-bench Verified
SWE-bench Verified is estimated from SWE-bench Pro with a linear curve fitted on 50 models measured on both: y = 0.8093·x + 0.3235, R² = 0.90, cross-validated error 3.7 pp. It is used for 18 estimates.
| Estimated model | SWE-bench Pro | SWE-bench Verified | Source |
|---|---|---|---|
| Claude Fable 5.1 | 81.2% | 98.1% | estimated ± 3.7 pp, medium confidence |
| Sakana Fugu-Ultra | 73.7% | 92.0% | estimated ± 3.7 pp, high confidence |
| Ling 3.1 Flash | 68.7% | 88.0% | estimated ± 3.7 pp, high confidence |
| Grok 4.5 | 64.7% | 84.7% | estimated ± 3.7 pp, high confidence |
| GPT-5.6 Sol | 64.6% | 84.6% | estimated ± 3.7 pp, high confidence |
| GPT-5.6 Terra | 63.4% | 83.7% | estimated ± 3.7 pp, high confidence |
| GPT-5.6 Luna | 62.7% | 83.1% | estimated ± 3.7 pp, high confidence |
| Muse Spark 1.1 | 61.5% | 82.1% | estimated ± 3.7 pp, high confidence |
| Sage-A1.5 | 61.2% | 81.9% | estimated ± 3.7 pp, high confidence |
| Atria Dawn Preview | 59.6% | 80.6% | estimated ± 3.7 pp, high confidence |
| Laguna S 2.1 | 59.4% | 80.4% | estimated ± 3.7 pp, high confidence |
| Sakana Fugu | 59.0% | 80.1% | estimated ± 3.7 pp, high confidence |
| MiMo-V2.5-Pro | 57.2% | 78.6% | estimated ± 3.7 pp, high confidence |
| Ling 3.0 Flash | 56.6% | 78.2% | estimated ± 3.7 pp, high confidence |
| Step 3.7 Flash | 56.3% | 77.9% | estimated ± 3.7 pp, high confidence |
| MiMo-V2.5 | 56.1% | 77.8% | estimated ± 3.7 pp, high confidence |
| Gemini 3.5 Flash | 55.1% | 76.9% | estimated ± 3.7 pp, high confidence |
| Gemini 3.5 Flash-Lite | 54.2% | 76.2% | estimated ± 3.7 pp, high confidence |