benchgap
Calibration

AIME 2024 → MATH-500

MATH-500 is estimated from AIME 2024 with a Michaelis–Menten curve fitted on 13 models measured on both: y = 1.0449·x / (0.06006 + x), R² = 0.96, cross-validated error 1.2 pp. It is used for 12 estimates.

Estimated modelAIME 2024MATH-500Source
Ripple-65B Preview97.1%98.4%estimated ± 1.2 pp, medium confidence
GPT-OSS 120B95.8%98.3%estimated ± 1.2 pp, medium confidence
GPT-OSS 20B92.1%98.1%estimated ± 1.2 pp, medium confidence
Gemini 2.5 Pro92.0%98.1%estimated ± 1.2 pp, medium confidence
DeepSeek-R1-052891.4%98.0%estimated ± 1.2 pp, medium confidence
Grok 3 Mini89.5%97.9%estimated ± 1.2 pp, medium confidence
o3-mini87.3%97.8%estimated ± 1.2 pp, medium confidence
Phi-4-reasoning-plus81.3%97.3%estimated ± 1.2 pp, high confidence
Phi-4-reasoning75.3%96.8%estimated ± 1.2 pp, high confidence
DeepSeek V3.166.3%95.8%estimated ± 1.2 pp, high confidence
Grok 3 [Beta]52.2%93.7%estimated ± 1.2 pp, high confidence
Claude 3.5 Haiku5.3%49.0%estimated ± 1.2 pp, medium confidence

Fitted on

ModelAIME 2024MATH-500
Claude 3.7 Sonnet23.3%82.2%
Claude 3.7 Sonnet Thinking61.3%96.2%
DeepSeek-R179.8%97.3%
DeepSeek V339.2%90.2%
Kimi K269.6%97.4%
Magistral Medium73.6%94.3%
Magistral Small70.7%95.9%
MiniMax M1 80k86.0%96.8%
Qwen3 235B-A22B40.1%91.2%
Qwen3 235B-A22B (Reasoning)85.7%98.0%
Qwen3 32B31.0%88.6%
Qwen3 32B (Reasoning)81.4%97.2%
QwQ-32B79.5%98.0%