benchgap
Calibration

FrontierMath v2 (Tiers 1-3) → FrontierMath v2 (Tier 4)

FrontierMath v2 (Tier 4) is estimated from FrontierMath v2 (Tiers 1-3) with a inverse Michaelis–Menten curve fitted on 56 models measured on both: y = 0.34536·(x − 0.2178) / (0.2178 + 0.9434 − x), R² = 0.99, cross-validated error 4.2 pp. It is used for 8 estimates.

Estimated modelFrontierMath v2 (Tiers 1-3)FrontierMath v2 (Tier 4)Source
o333.3%4.8%estimated ± 4.2 pp, high confidence
Gemini 3.1 Flash-Lite27.7%2.3%estimated ± 4.2 pp, high confidence
GPT-4.16.0%0.0%estimated ± 4.2 pp, medium confidence
GPT-4.1 mini6.7%0.0%estimated ± 4.2 pp, medium confidence
GPT-4o0.4%0.0%estimated ± 4.2 pp, medium confidence
o114.7%0.0%estimated ± 4.2 pp, medium confidence
Qwen3.5 Flash18.2%0.0%estimated ± 4.2 pp, medium confidence
Qwen3.7 Flash19.3%0.0%estimated ± 4.2 pp, medium confidence

Fitted on

ModelFrontierMath v2 (Tiers 1-3)FrontierMath v2 (Tier 4)
Claude Fable 587.0%90.2%
Claude Fable 5.190.2%87.8%
Claude Fable 5.295.1%100.0%
Claude Haiku 5.575.1%46.3%
Claude Opus 4.534.4%4.9%
Claude Opus 4.666.0%26.8%
Claude Opus 4.770.2%31.7%
Claude Opus 4.880.0%56.1%
Claude Opus 585.6%73.2%
Claude Opus 5.591.2%95.0%
Claude Sonnet 4.523.9%2.4%
Claude Sonnet 565.6%29.3%
Claude Sonnet 5.588.8%80.5%
DeepSeek V4.1 Flash 101767.4%26.8%
DeepSeek V4 Pro64.6%26.8%
Gemini 2.5 Pro24.6%0.0%
Gemini 3.1 Pro59.6%26.8%
Gemini 3.5 Flash62.8%26.8%
Gemini 3.5 Flash-Lite26.0%0.0%
Gemini 3.6 Flash58.9%22.0%
Gemini 3.7 Flash71.6%36.6%
Gemini 3.8 Flash68.4%22.0%
Gemini 3 Flash51.2%17.1%
Gemini 4 Argon86.0%78.0%
Gemini 4 Carbon90.5%95.1%
GLM-5.259.2%29.3%
GLM-5.368.8%29.3%
GLM-5.3-Flash55.8%17.1%
GPT-5.267.4%31.7%
GPT-5.478.6%49.0%
GPT-5.4 mini51.2%9.8%
GPT-5.4 nano44.9%12.2%
GPT-5.4 Pro82.5%58.5%
GPT-5.585.3%72.5%
GPT-5.5 Pro87.7%78.0%
GPT-5.6 Luna78.6%58.5%
GPT-5.6 Sol89.0%83.0%
GPT-5.6 Terra84.9%68.3%
GPT-6.1 Sol93.7%100.0%
GPT-6.1 Terra90.9%92.7%
GPT-6.2 Astra92.6%100.0%
GPT-6 Astra93.7%97.6%
GPT-6 Luna78.9%56.1%
GPT-6 Sol89.8%90.0%
Grok 4.342.8%14.6%
Grok 4.557.2%24.4%
Grok 4.666.0%31.7%
Grok 4.753.0%17.1%
Inkling33.3%4.9%
Inkling-Small46.3%17.1%
Kimi K2.657.2%25.6%
Kimi K2.7 Code54.0%12.2%
Muse Spark 1.374.0%46.3%
o4-mini36.1%4.9%
Qwen3.7 Max64.6%34.1%
Qwen3.8 Max74.7%46.3%