Calibration
Terminal-Bench 4.0 → Terminal-Bench 2.1 (Vals AI)
Terminal-Bench 2.1 (Vals AI) is estimated from Terminal-Bench 4.0 with a Hill curve fitted on 5 models measured on both: y = 0.7576 + (1.2000 − 0.7576)·x^6.00 / (0.70190^6.00 + x^6.00), R² = 0.65, cross-validated error 5.4 pp. It is used for 15 estimates.
| Estimated model | Terminal-Bench 4.0 | Terminal-Bench 2.1 (Vals AI) | Source |
|---|---|---|---|
| Claude Sonnet 5.5 | 70.6% | 98.3% | estimated ± 5.4 pp, low confidence |
| Claude Opus 5.5 | 66.4% | 94.2% | estimated ± 5.4 pp, low confidence |
| Kimi K3.1 Pro | 65.7% | 93.5% | estimated ± 5.4 pp, low confidence |
| Claude Mythos 5.1 | 60.9% | 89.0% | estimated ± 5.4 pp, low confidence |
| Gemini 4 Argon | 57.4% | 85.9% | estimated ± 5.4 pp, low confidence |
| Pareto 26.9 | 51.0% | 81.4% | estimated ± 5.4 pp, low confidence |
| Pareto 26.10 Preview | 50.8% | 81.3% | estimated ± 5.4 pp, low confidence |
| GPT-6.1 Terra | 50.3% | 81.0% | estimated ± 5.4 pp, low confidence |
| Ling 3.1 Flash | 46.0% | 79.0% | estimated ± 5.4 pp, low confidence |
| Claude Haiku 5.5 | 39.2% | 77.1% | estimated ± 5.4 pp, low confidence |
| MiniMax M3.1 Flash | 38.1% | 76.9% | estimated ± 5.4 pp, low confidence |
| MiMo-V2.6-Pro | 34.9% | 76.4% | estimated ± 5.4 pp, low confidence |
| Step 5 Preview | 33.3% | 76.3% | estimated ± 5.4 pp, low confidence |
| MiMo-V2.6-Flash | 28.8% | 76.0% | estimated ± 5.4 pp, low confidence |
| SWE-2 | 27.3% | 75.9% | estimated ± 5.4 pp, low confidence |
Fitted on
| Model | Terminal-Bench 4.0 | Terminal-Bench 2.1 (Vals AI) |
|---|---|---|
| Claude Fable 5.1 | 55.8% | 85.0% |
| DeepSeek V4.1 Flash 1017 | 42.3% | 74.5% |
| Gemini 3.8 Flash | 19.1% | 81.3% |
| GPT-6 Astra | 57.9% | 87.3% |
| Grok 4.7 | 38.0% | 73.4% |