Calibration
Vals MMLU-Pro → AA MMLU-Pro
AA MMLU-Pro is estimated from Vals MMLU-Pro with a Michaelis–Menten + offset curve fitted on 10 models measured on both: y = 0.2649 + 2.0000·x / (1.97410 + x), R² = 0.68, cross-validated error 1.1 pp. It is used for 13 estimates.
| Estimated model | Vals MMLU-Pro | AA MMLU-Pro | Source |
|---|---|---|---|
| Gemini 3.1 Pro | 91.0% | 89.6% | estimated ± 1.1 pp, medium confidence |
| Qwen3.8 Max | 88.6% | 88.4% | estimated ± 1.1 pp, high confidence |
| Claude Sonnet 4.5 Thinking | 87.4% | 87.9% | estimated ± 1.1 pp, high confidence |
| Grok 4.20 | 86.3% | 87.3% | estimated ± 1.1 pp, high confidence |
| Gemini 3.1 Flash-Lite | 86.2% | 87.3% | estimated ± 1.1 pp, high confidence |
| GLM-5.3-Flash | 86.1% | 87.2% | estimated ± 1.1 pp, high confidence |
| Qwen3.5 Flash | 84.1% | 86.2% | estimated ± 1.1 pp, high confidence |
| MiMo-V2.5 | 82.9% | 85.6% | estimated ± 1.1 pp, high confidence |
| GLM-4.5 | 81.2% | 84.8% | estimated ± 1.1 pp, high confidence |
| Claude Haiku 4.5 | 78.7% | 83.5% | estimated ± 1.1 pp, medium confidence |
| o3-mini | 78.7% | 83.5% | estimated ± 1.1 pp, medium confidence |
| Claude 3.5 Sonnet | 78.4% | 83.3% | estimated ± 1.1 pp, medium confidence |
| Gemini 1.5 Pro | 75.3% | 81.7% | estimated ± 1.1 pp, medium confidence |
Fitted on
| Model | Vals MMLU-Pro | AA MMLU-Pro |
|---|---|---|
| Claude Opus 4.5 Thinking | 87.3% | 89.5% |
| Gemini 3 Pro | 90.1% | 89.8% |
| GLM-4.7 | 82.7% | 85.6% |
| GPT-5 | 86.5% | 87.1% |
| GPT-5.1 | 86.4% | 87.0% |
| GPT-5.2 | 86.2% | 87.4% |
| Grok 4 | 85.3% | 86.6% |
| Grok 4.1 Fast (Reasoning) | 84.2% | 85.4% |
| Grok 4 Fast (Reasoning) | 79.7% | 85.0% |
| o3 | 85.6% | 85.3% |