Calibration
Vals SWE-bench → CursorBench 3.1
CursorBench 3.1 is estimated from Vals SWE-bench with a linear curve fitted on 7 models measured on both: y = 1.0263·x + -0.2797, R² = 0.68, cross-validated error 5.8 pp. It is used for 9 estimates.
| Estimated model | Vals SWE-bench | CursorBench 3.1 | Source |
|---|---|---|---|
| GLM-5.3-Flash | 92.0% | 66.4% | estimated ± 5.8 pp, low confidence |
| Claude Opus 4.7 | 82.0% | 56.2% | estimated ± 5.8 pp, low confidence |
| Gemini 3 Flash | 75.0% | 49.0% | estimated ± 5.8 pp, low confidence |
| GPT-5.2-Codex | 72.4% | 46.3% | estimated ± 5.8 pp, low confidence |
| GLM-4.7 | 69.4% | 43.3% | estimated ± 5.8 pp, low confidence |
| Claude Haiku 4.5 | 66.6% | 40.4% | estimated ± 5.8 pp, low confidence |
| Mistral Medium 3.5 128B | 66.4% | 40.2% | estimated ± 5.8 pp, low confidence |
| Qwen3.5 Flash | 64.4% | 38.1% | estimated ± 5.8 pp, low confidence |
| Gemini 3.1 Flash-Lite | 62.8% | 36.5% | estimated ± 5.8 pp, low confidence |
Fitted on
| Model | Vals SWE-bench | CursorBench 3.1 |
|---|---|---|
| Claude Fable 5 | 95.0% | 70.6% |
| Claude Opus 4.8 | 88.6% | 58.4% |
| Claude Sonnet 4.6 | 77.4% | 48.8% |
| Composer 2.5 | 79.6% | 63.2% |
| Gemini 3.5 Flash | 78.8% | 49.8% |
| GPT-5.5 | 82.6% | 59.2% |
| Kimi K2.6 | 76.2% | 47.6% |