Calibration
AIME 2024 → MATH-500
MATH-500 is estimated from AIME 2024 with a Michaelis–Menten curve fitted on 13 models measured on both: y = 1.0449·x / (0.06006 + x), R² = 0.96, cross-validated error 1.2 pp. It is used for 12 estimates.
| Estimated model | AIME 2024 | MATH-500 | Source |
|---|---|---|---|
| Ripple-65B Preview | 97.1% | 98.4% | estimated ± 1.2 pp, medium confidence |
| GPT-OSS 120B | 95.8% | 98.3% | estimated ± 1.2 pp, medium confidence |
| GPT-OSS 20B | 92.1% | 98.1% | estimated ± 1.2 pp, medium confidence |
| Gemini 2.5 Pro | 92.0% | 98.1% | estimated ± 1.2 pp, medium confidence |
| DeepSeek-R1-0528 | 91.4% | 98.0% | estimated ± 1.2 pp, medium confidence |
| Grok 3 Mini | 89.5% | 97.9% | estimated ± 1.2 pp, medium confidence |
| o3-mini | 87.3% | 97.8% | estimated ± 1.2 pp, medium confidence |
| Phi-4-reasoning-plus | 81.3% | 97.3% | estimated ± 1.2 pp, high confidence |
| Phi-4-reasoning | 75.3% | 96.8% | estimated ± 1.2 pp, high confidence |
| DeepSeek V3.1 | 66.3% | 95.8% | estimated ± 1.2 pp, high confidence |
| Grok 3 [Beta] | 52.2% | 93.7% | estimated ± 1.2 pp, high confidence |
| Claude 3.5 Haiku | 5.3% | 49.0% | estimated ± 1.2 pp, medium confidence |
Fitted on
| Model | AIME 2024 | MATH-500 |
|---|---|---|
| Claude 3.7 Sonnet | 23.3% | 82.2% |
| Claude 3.7 Sonnet Thinking | 61.3% | 96.2% |
| DeepSeek-R1 | 79.8% | 97.3% |
| DeepSeek V3 | 39.2% | 90.2% |
| Kimi K2 | 69.6% | 97.4% |
| Magistral Medium | 73.6% | 94.3% |
| Magistral Small | 70.7% | 95.9% |
| MiniMax M1 80k | 86.0% | 96.8% |
| Qwen3 235B-A22B | 40.1% | 91.2% |
| Qwen3 235B-A22B (Reasoning) | 85.7% | 98.0% |
| Qwen3 32B | 31.0% | 88.6% |
| Qwen3 32B (Reasoning) | 81.4% | 97.2% |
| QwQ-32B | 79.5% | 98.0% |