benchgap
Calibration

GPQA Diamond → CritPt

CritPt is estimated from GPQA Diamond with a inverse Michaelis–Menten curve fitted on 52 models measured on both: y = 0.46832·(x − 0.8394) / (0.8394 + 0.3034 − x), R² = 0.80, cross-validated error 4.6 pp. It is used for 8 estimates.

Estimated modelGPQA DiamondCritPtSource
Pareto 26.10 Preview92.4%18.1%estimated ± 4.6 pp, high confidence
Particle-190.4%12.7%estimated ± 4.6 pp, high confidence
Mercury 389.2%9.9%estimated ± 4.6 pp, high confidence
Nemotron 3.5 Bolt 74B-A9B BF1688.8%9.0%estimated ± 4.6 pp, high confidence
Grok 4.2088.5%8.3%estimated ± 4.6 pp, high confidence
OrcaCyber-Nano-1.088.3%7.8%estimated ± 4.6 pp, high confidence
LLaDA2.2-mini44.4%0.0%estimated ± 4.6 pp, medium confidence
Mellum2.1-12B-A2.5B-Thinking64.6%0.0%estimated ± 4.6 pp, high confidence

Fitted on

ModelGPQA DiamondCritPt
A.X K285.6%8.9%
Claude Opus 4.689.2%2.8%
Claude Opus 4.794.2%12.0%
Claude Opus 4.893.6%20.9%
DeepSeek V4.1 Flash 101790.9%14.3%
DeepSeek V4 Flash 073188.1%16.6%
DeepSeek V4 Pro90.1%18.0%
Gemini 3.1 Pro94.3%17.7%
Gemini 3.5 Flash92.7%13.1%
Gemma 4 12B78.8%0.0%
GLM-586.0%2.0%
GLM-5.186.2%4.6%
GLM-5.291.2%20.9%
GPT-5.492.8%23.4%
GPT-5.593.6%27.1%
GPT-5.6 Luna92.3%20.6%
GPT-5.6 Sol94.6%32.3%
GPT-5.6 Terra92.9%30.0%
GPT-6 Astra96.0%31.7%
Hy3 Preview87.2%4.9%
Hy4 preview92.3%16.9%
Inkling87.9%5.4%
Inkling-Small89.5%8.3%
K-EXAONE 2.082.2%0.9%
Kimi K2.690.5%8.0%
Kimi K3.1 Pro94.2%24.6%
Kimi K2.587.6%3.1%
Kimi K393.5%23.4%
Ling 3.0 Flash85.0%1.7%
Mercury 2.579.0%0.0%
MiniCPM5-2B70.2%0.3%
MiniMax M2.787.0%0.6%
Muse Spark89.5%11.3%
Nemotron 3.5 Bolt 74B-A9B86.4%4.3%
Nemotron 3.5 Lightning 30B A3B NVFP475.6%0.0%
Nemotron 3.5 Super VL 120B A12B89.5%10.9%
Nemotron 3.5 Super VL 120B A12B NVFP489.9%10.6%
Nemotron 3 Nano Omni 30B A3B72.2%0.0%
Nemotron 3 Ultra87.0%3.1%
Qwen3.7 Max92.4%13.4%
Qwen3.7 Plus90.3%9.1%
Qwen3.8-27B89.2%5.4%
Qwen3.8-Flash-Next91.7%11.1%
Beam 1.0 501B-A23B90.6%16.3%
Beam 1.0 501B-A23B FP890.3%17.4%
Beam 1.0 501B-A23B NVFP489.3%15.4%
Sarvam 105B78.7%0.0%
Sarvam 30B66.5%0.3%
Solar Pro 489.0%6.3%
Step 5 Preview93.5%20.9%
Trinity-Large-Preview63.3%0.9%
Trinity-Large-Thinking76.3%0.9%