benchgap
Calibration

GPQA → CritPt

CritPt is estimated from GPQA with a offset logistic curve fitted on 84 models measured on both: y = 0.0033 + (0.4421 − 0.0033) / (1 + exp(−40.35·(x − 0.9360))), R² = 0.82, cross-validated error 4.0 pp. It is used for 42 estimates.

Estimated modelGPQACritPtSource
GPT-6.2 Astra97.0%35.3%estimated ± 4.0 pp, medium confidence
GPT-6.2 Sol96.0%32.1%estimated ± 4.0 pp, high confidence
Sakana Fugu95.5%30.3%estimated ± 4.0 pp, high confidence
Sakana Fugu-Ultra95.5%30.3%estimated ± 4.0 pp, high confidence
Claude Mythos 594.1%24.5%estimated ± 4.0 pp, high confidence
Ornith-1.5-397B92.8%18.8%estimated ± 4.0 pp, high confidence
Qwen3.8 Max92.6%17.9%estimated ± 4.0 pp, high confidence
Qwen3.8-Omni-Flash91.0%11.7%estimated ± 4.0 pp, high confidence
Ornith-1.5-35B-A3B89.2%6.7%estimated ± 4.0 pp, high confidence
Ornith-1.5-9B86.4%2.6%estimated ± 4.0 pp, high confidence
Ternary Bonsai 2 27B85.8%2.1%estimated ± 4.0 pp, high confidence
MAI-Thinking-184.2%1.3%estimated ± 4.0 pp, high confidence
Ling 3.0 Flash FP884.0%1.2%estimated ± 4.0 pp, high confidence
Claude Sonnet 4.583.4%1.0%estimated ± 4.0 pp, high confidence
DeepSeek-R1-052881.0%0.6%estimated ± 4.0 pp, high confidence
Grok 3 Mini80.3%0.5%estimated ± 4.0 pp, high confidence
Nemotron 3 Super 100B79.2%0.5%estimated ± 4.0 pp, high confidence
o1-pro79.0%0.5%estimated ± 4.0 pp, high confidence
Qwen3 235B 250777.5%0.4%estimated ± 4.0 pp, high confidence
Grok 3 [Beta]75.4%0.4%estimated ± 4.0 pp, high confidence
GLM-4.7-Flash75.2%0.4%estimated ± 4.0 pp, high confidence
LFM2.5-24B-A2B74.4%0.3%estimated ± 4.0 pp, high confidence
ZAYA1-8B71.0%0.3%estimated ± 4.0 pp, high confidence
Spark-X2.6-4B69.9%0.3%estimated ± 4.0 pp, high confidence
GPT-4.569.5%0.3%estimated ± 4.0 pp, high confidence
Phi-4-reasoning-plus68.9%0.3%estimated ± 4.0 pp, high confidence
LFM3-1.7B63.4%0.3%estimated ± 4.0 pp, high confidence
Qwen2.5-Max60.1%0.3%estimated ± 4.0 pp, high confidence
Claude 3.5 Sonnet59.4%0.3%estimated ± 4.0 pp, high confidence
Mellum2-12B-A2.5B-Thinking57.6%0.3%estimated ± 4.0 pp, high confidence
ZAYA1-74B-Preview57.3%0.3%estimated ± 4.0 pp, high confidence
Grok 256.0%0.3%estimated ± 4.0 pp, high confidence
Grok 2 mini51.0%0.3%estimated ± 4.0 pp, high confidence
Gemini 2.0 Flash-Lite50.5%0.3%estimated ± 4.0 pp, high confidence
Qwen2.5-VL-32B46.0%0.3%estimated ± 4.0 pp, high confidence
Soofi S 30B-A3B43.4%0.3%estimated ± 4.0 pp, high confidence
Mellum2-12B-A2.5B-Instruct40.9%0.3%estimated ± 4.0 pp, high confidence
LFM2.5-350M30.6%0.3%estimated ± 4.0 pp, high confidence
Phi-3.5-mini30.4%0.3%estimated ± 4.0 pp, high confidence
Granite-4.0-1B29.9%0.3%estimated ± 4.0 pp, high confidence
LFM2.5-VL-450M25.7%0.3%estimated ± 4.0 pp, high confidence
LFM2.5-230M25.4%0.3%estimated ± 4.0 pp, high confidence

Fitted on

ModelGPQACritPt
Claude 3.5 Haiku41.6%0.0%
Claude 3.7 Sonnet68.0%0.0%
Claude 3.7 Sonnet Thinking78.2%0.9%
Claude Opus 4.587.0%0.3%
Claude Opus 4.691.3%2.8%
Claude Opus 4.794.2%12.0%
Claude Opus 4.893.6%20.9%
Claude Sonnet 4.689.9%0.9%
DeepSeek-R171.5%1.4%
DeepSeek V359.1%0.0%
DeepSeek V3.174.9%0.0%
DeepSeek V4.1 Flash 101790.9%14.3%
DeepSeek V4 Flash 073188.1%16.6%
DeepSeek V4 Pro90.1%18.0%
Gemini 2.0 Flash65.2%0.0%
Gemini 2.5 Flash-Lite64.6%0.0%
Gemini 2.5 Pro83.0%2.6%
Gemini 3.5 Flash92.2%13.1%
Gemma 4 12B78.8%0.0%
Gemma 4 26B A4B82.3%0.0%
Gemma 4 31B84.3%1.4%
Gemma 4 E2B43.4%0.0%
Gemma 4 E4B58.6%0.6%
GLM-4.785.7%1.7%
GLM-586.0%2.0%
GLM-5.291.2%20.9%
GPT-4.166.3%0.0%
GPT-4.1 mini64.2%0.0%
GPT-4.1 nano50.3%0.0%
GPT-5.292.4%11.6%
GPT-5.492.8%23.4%
GPT-5.4 mini88.0%10.0%
GPT-5.4 nano82.8%9.3%
GPT-5.593.6%27.1%
GPT-5.6 Luna92.3%20.6%
GPT-5.6 Sol94.6%32.3%
GPT-5.6 Terra92.9%30.0%
GPT-6 Astra96.0%31.7%
GPT-OSS 120B80.1%1.1%
GPT-OSS 20B71.5%1.4%
Granite-4.0-350M24.1%0.0%
Granite-4.0-H-1B29.7%0.0%
Granite-4.0-H-350M26.1%0.0%
Granite 4.2 30B66.4%0.3%
Granite 4.2 3B54.8%0.0%
Granite 4.2 8B64.1%0.3%
Grok 4.390.1%8.0%
Hy3 Preview87.2%4.9%
Hy4 preview92.3%16.9%
Inkling87.9%5.4%
Inkling-Small89.5%8.3%
Kimi K2.690.5%8.0%
Kimi K3.1 Pro94.2%24.6%
Kimi K275.1%0.0%
Kimi K2.587.6%3.1%
Kimi K393.5%23.4%
Ling 2.6 Flash59.0%0.0%
Ling 3.0 Flash85.0%1.7%
Llama 3.1 70B46.7%0.0%
Llama 3.3 70B50.5%0.0%
Magistral Medium70.8%0.3%
Magistral Small68.2%0.0%
MiMo-V2-Flash83.7%0.0%
Mistral Small 3.146.0%0.0%
Nemotron 3.5 Lightning 30B A3B NVFP475.6%0.0%
Nemotron 3 Nano 30B73.0%0.0%
Nemotron 3 Nano Omni 30B A3B72.2%0.0%
Nemotron 3 Ultra87.0%3.1%
Nemotron Ultra 253B76.0%0.0%
o175.7%0.3%
o4-mini77.6%0.6%
Phi-456.1%0.0%
Phi-4-reasoning65.8%0.0%
Qwen3.5-122B-A10B86.6%0.6%
Qwen3.5-27B85.5%0.9%
Qwen3.5-35B-A3B84.2%0.9%
Qwen3.5 397B88.4%1.7%
Qwen3.6-27B87.8%1.1%
Qwen3.6-35B-A3B86.0%0.3%
Qwen3.6 Plus90.4%2.9%
Qwen3.7 Max92.4%13.4%
Qwen3.7 Plus90.3%9.1%
Qwen3.8-27B89.2%5.4%
Qwen3.8-Flash-Next91.7%11.1%