Calibration
Vals SWE-bench → AA Coding Index
AA Coding Index is estimated from Vals SWE-bench with a Hill curve fitted on 44 models measured on both: y = 0.2185 + (0.8470 − 0.2185)·x^6.00 / (0.70198^6.00 + x^6.00), R² = 0.74, cross-validated error 5.8 pp. It is used for 5 estimates.
| Estimated model | Vals SWE-bench | AA Coding Index | Source |
|---|---|---|---|
| Gemini 3.1 Flash-Lite | 62.8% | 43.1% | estimated ± 5.8 pp, medium confidence |
| Gemini 3 Flash | 75.0% | 59.4% | estimated ± 5.8 pp, medium confidence |
| GPT-5.2-Codex | 72.4% | 56.2% | estimated ± 5.8 pp, medium confidence |
| MiMo-V2.5 | 71.0% | 54.3% | estimated ± 5.8 pp, medium confidence |
| Qwen3.5 Flash | 64.4% | 45.3% | estimated ± 5.8 pp, medium confidence |