Calibration
Vals LiveCodeBench → SWE-bench Pro
SWE-bench Pro is estimated from Vals LiveCodeBench with a offset logistic curve fitted on 32 models measured on both: y = 0.5697 + (0.8227 − 0.5697) / (1 + exp(−200.00·(x − 0.8833))), R² = 0.69, cross-validated error 4.8 pp. It is used for 11 estimates.
| Estimated model | Vals LiveCodeBench | SWE-bench Pro | Source |
|---|---|---|---|
| Gemini 3.7 Flash | 88.7% | 74.1% | estimated ± 4.8 pp, high confidence |
| GPT-5.2-Codex | 88.0% | 65.6% | estimated ± 4.8 pp, high confidence |
| Gemini 3 Flash | 85.6% | 57.1% | estimated ± 4.8 pp, high confidence |
| GPT-5.1-Codex | 85.6% | 57.1% | estimated ± 4.8 pp, high confidence |
| Claude Opus 4.7 | 85.1% | 57.0% | estimated ± 4.8 pp, high confidence |
| Grok 4.3 | 84.5% | 57.0% | estimated ± 4.8 pp, high confidence |
| GPT-5.1-Codex-Max | 83.6% | 57.0% | estimated ± 4.8 pp, high confidence |
| Qwen3.5 Flash | 83.3% | 57.0% | estimated ± 4.8 pp, high confidence |
| GLM-4.6 | 81.0% | 57.0% | estimated ± 4.8 pp, high confidence |
| Gemini 3.1 Flash-Lite | 80.1% | 57.0% | estimated ± 4.8 pp, high confidence |
| GLM-4.5 | 67.4% | 57.0% | estimated ± 4.8 pp, medium confidence |