benchgap
Calibration

Vals GPQA Diamond → AA-GPQA Diamond

AA-GPQA Diamond is estimated from Vals GPQA Diamond with a offset logistic curve fitted on 46 models measured on both: y = 0.6876 + (0.9519 − 0.6876) / (1 + exp(−19.21·(x − 0.8131))), R² = 0.79, cross-validated error 3.9 pp. It is used for 8 estimates.

Estimated modelVals GPQA DiamondAA-GPQA DiamondSource
GLM-5.3-Flash86.4%88.0%estimated ± 3.9 pp, high confidence
Qwen3.5 Flash82.8%83.9%estimated ± 3.9 pp, high confidence
MiMo-V2.581.6%82.3%estimated ± 3.9 pp, high confidence
Gemini 3.1 Flash-Lite81.1%81.7%estimated ± 3.9 pp, high confidence
Claude Haiku 4.572.2%72.7%estimated ± 3.9 pp, high confidence
GLM-4.572.2%72.7%estimated ± 3.9 pp, high confidence
Laguna XS.255.1%68.9%estimated ± 3.9 pp, high confidence
Laguna M.127.0%68.8%estimated ± 3.9 pp, medium confidence

Fitted on

ModelVals GPQA DiamondAA-GPQA Diamond
Claude Fable 593.2%92.6%
Claude Fable 5.193.4%93.7%
Claude Opus 4.790.2%88.5%
Claude Opus 4.892.4%92.0%
Claude Opus 593.4%93.2%
Claude Sonnet 4.685.6%79.9%
Claude Sonnet 588.9%91.1%
DeepSeek V4 Flash 073189.9%90.8%
DeepSeek V4 Pro 081392.4%92.8%
Gemini 3.1 Pro95.5%94.1%
Gemini 3.5 Flash92.7%92.2%
Gemini 3.5 Flash-Lite83.8%83.8%
Gemini 3.6 Flash93.4%92.8%
Gemini 3.7 Flash93.9%94.5%
Gemini 3.8 Flash94.4%95.3%
Gemini 3 Flash87.9%81.2%
GLM-4.674.5%63.2%
GLM-4.780.1%85.9%
GLM-5.184.5%86.8%
GLM-5.285.6%89.5%
GLM-5.388.1%91.7%
GPT-5.4 mini83.1%87.5%
GPT-5.4 nano77.5%81.7%
GPT-5.593.2%93.5%
GPT-5.6 Luna91.7%91.1%
GPT-5.6 Sol95.2%94.1%
GPT-5.6 Terra90.9%92.5%
Grok 4.391.4%90.1%
Grok 4.592.9%93.1%
Grok 4.694.7%94.9%
Inkling87.1%87.2%
Inkling-Small83.6%89.5%
Kimi K2.689.1%91.1%
Kimi K392.9%93.5%
Ling 3.0 Flash84.8%85.5%
MiMo-V2.5-Pro82.6%86.6%
MiMo-V2-Flash59.3%65.6%
MiniMax M2.786.6%87.4%
MiniMax M392.7%92.9%
Mistral Medium 3.5 128B34.8%74.8%
Muse Spark89.6%88.4%
Muse Spark 1.191.2%89.8%
Nemotron 3 Ultra86.1%86.7%
Qwen3.6 Plus87.4%88.2%
Qwen3.7 Max90.2%92.3%
Qwen3.8-27B88.9%90.5%