benchgap
Calibration

AA-HLE → CritPt

CritPt is estimated from AA-HLE with a offset logistic curve fitted on 200 models measured on both: y = 0.0002 + (0.3492 − 0.0002) / (1 + exp(−14.18·(x − 0.4275))), R² = 0.92, cross-validated error 3.4 pp. It is used for 9 estimates.

Estimated modelAA-HLECritPtSource
Inkling 2 Small44.1%19.1%estimated ± 3.4 pp, high confidence
Phi-4 Multimodal Instruct5.0%0.2%estimated ± 3.4 pp, high confidence
DeepSeek R1 Distill Qwen 32B4.6%0.2%estimated ± 3.4 pp, high confidence
Gemini 1.5 Pro4.6%0.2%estimated ± 3.4 pp, high confidence
Gemini 1.0 Pro4.2%0.2%estimated ± 3.4 pp, high confidence
Qwen2.5 Coder 32B Instruct3.5%0.2%estimated ± 3.4 pp, high confidence
Gemini 1.5 Flash3.2%0.1%estimated ± 3.4 pp, high confidence
GPT-4 Turbo3.1%0.1%estimated ± 3.4 pp, high confidence
Claude 3 Opus2.8%0.1%estimated ± 3.4 pp, high confidence

Fitted on

ModelAA-HLECritPt
A.X K229.6%8.9%
Apodex 1.134.1%4.6%
Apodex 1.1 Mini34.1%4.6%
Celeris-16.8%0.0%
Claude 3.5 Haiku3.6%0.0%
Claude 3.7 Sonnet4.2%0.0%
Claude 3.7 Sonnet Thinking9.7%0.9%
Claude 3 Haiku4.1%0.0%
Claude 4 Sonnet4.3%1.1%
Claude Fable 555.5%28.6%
Claude Fable 5.159.1%29.7%
Claude Fable 5.274.3%35.4%
Claude Haiku 5.544.4%18.9%
Claude Opus 4.513.2%0.3%
Claude Opus 4.5 Thinking30.1%4.6%
Claude Opus 4.619.1%2.8%
Claude Opus 4.6 (Adaptive)39.9%12.6%
Claude Opus 4.742.3%12.0%
Claude Opus 4.848.7%20.9%
Claude Opus 554.9%29.1%
Claude Opus 5.561.4%31.7%
Claude Sonnet 4.613.3%0.9%
Claude Sonnet 541.3%16.9%
Claude Sonnet 5.555.0%31.4%
Command A+12.0%0.3%
DeepSeek-R115.8%1.4%
DeepSeek V32.9%0.0%
DeepSeek V3 03244.7%0.0%
DeepSeek V3.16.7%0.0%
DeepSeek V3.211.2%0.9%
DeepSeek V4.1 Flash 101739.2%14.3%
DeepSeek V4 Flash 073138.6%16.6%
DeepSeek V4 Pro41.0%18.0%
Devstral Small3.8%0.0%
Exaone 4.0 1.2B5.7%0.0%
Exaone 4.0 32B5.0%0.0%
Gemini 2.0 Flash4.3%0.0%
Gemini 2.5 Flash4.7%1.4%
Gemini 2.5 Flash-Lite3.7%0.0%
Gemini 2.5 Pro22.5%2.6%
Gemini 3.1 Pro47.0%17.7%
Gemini 3.5 Flash42.7%13.1%
Gemini 3.5 Flash-Lite18.8%0.0%
Gemini 3.6 Flash40.8%10.6%
Gemini 3.7 Flash47.9%14.3%
Gemini 3.8 Flash47.8%18.3%
Gemini 3 Flash15.0%1.4%
Gemini 3 Pro39.7%9.1%
Gemini 4 Argon57.1%27.1%
Gemma 3 27B4.4%0.0%
Gemma 4 12B15.7%0.0%
Gemma 4 26B A4B19.3%0.0%
Gemma 4 31B23.6%1.4%
Gemma 4 E2B4.8%0.0%
Gemma 4 E4B3.8%0.6%
GLM-4.5-Air7.0%0.0%
GLM-4.65.5%0.0%
GLM-4.727.4%1.7%
GLM-529.3%2.0%
GLM-5.130.1%4.6%
GLM-5.241.1%20.9%
GLM-5.342.3%19.1%
GLM-5-Turbo27.8%0.3%
GLM-5V-Turbo17.1%0.6%
GPT-4.14.2%0.0%
GPT-4.1 mini5.0%0.0%
GPT-4.1 nano3.8%0.0%
GPT-4o2.4%0.0%
GPT-528.5%5.7%
GPT-5.128.5%4.9%
GPT-5.1-Codex25.7%5.7%
GPT-5.1-Codex-Max25.7%5.7%
GPT-5.237.7%11.6%
GPT-5.2-Codex35.7%8.7%
GPT-5.3 Codex42.5%16.9%
GPT-5.443.7%23.4%
GPT-5.4 mini28.1%10.0%
GPT-5.4 nano28.3%9.3%
GPT-5.545.8%27.1%
GPT-5.6 Luna39.5%20.6%
GPT-5.6 Sol49.5%32.3%
GPT-5.6 Terra42.9%30.0%
GPT-6.1 Sol52.9%31.7%
GPT-6.1 Terra46.2%28.0%
GPT-6 Astra54.7%31.7%
GPT-6 Luna38.5%19.4%
GPT-6 Sol47.9%30.9%
GPT-OSS 120B19.6%1.1%
GPT-OSS 20B11.0%1.4%
Granite-4.0-350M5.5%0.0%
Granite-4.0-H-1B5.0%0.0%
Granite-4.0-H-350M6.4%0.0%
Granite 4.2 30B11.2%0.3%
Granite 4.2 3B6.6%0.0%
Granite 4.2 8B9.7%0.3%
Grok 426.7%2.0%
Grok 4.1 Fast5.1%0.0%
Grok 4.1 Fast (Reasoning)19.3%2.9%
Grok 4.337.2%8.0%
Grok 4.542.7%15.4%
Grok 4.642.9%17.1%
Grok 4.743.1%17.7%
Grok 4 Fast (Reasoning)19.1%2.9%
Grok Code Fast 18.0%0.0%
Hy333.5%4.9%
Hy3 Preview33.5%4.9%
Inkling31.9%5.4%
Inkling-Small33.3%8.3%
K-Exaone13.9%1.1%
K-EXAONE 2.018.6%0.9%
Kimi K2.637.5%8.0%
Kimi K3.1 Pro52.5%24.6%
Kimi K27.4%0.0%
Kimi K2.530.7%3.1%
Kimi K2.7 Code35.0%10.0%
Kimi K346.9%23.4%
LFM2.5-2.6B6.2%0.0%
LFM2.5-8B-A1B6.9%0.0%
LFM2.5-VL-1.6B-Extract5.1%0.0%
Ling 2.6 Flash6.3%0.0%
Ling 3.0 Flash23.7%1.7%
Ling 3.0 Flash VL22.0%2.0%
Ling 3.0 Tiny9.3%0.0%
Ling 3.1 Flash39.4%18.0%
Llama 3.1 405B4.0%0.0%
Llama 3.1 70B3.6%0.0%
Llama 3.3 70B3.6%0.0%
Llama 4 Maverick4.9%0.0%
Llama 4 Scout3.8%0.0%
Magistral Medium9.8%0.3%
Magistral Small7.6%0.0%
Mercury 2.511.8%0.0%
MiMo-V2.5-Pro35.7%4.0%
MiMo-V2.6-Flash35.1%12.0%
MiMo-V2.6-Pro49.4%26.6%
MiMo-V2-Flash8.6%0.0%
MiMo-V2-Omni22.1%1.1%
MiMo-V2-Pro30.4%0.3%
MiniCPM5-2B8.9%0.3%
MiniMax M2.729.6%0.6%
MiniMax M339.0%3.7%
MiniMax M3.1 Flash48.0%22.3%
Mistral Large 23.3%0.0%
Mistral Large 34.2%0.0%
Mistral Medium 34.1%0.0%
Mistral Medium 3.5 128B13.8%0.0%
Mistral Small 3.14.3%0.0%
Mistral Small 49.9%0.3%
Muse Glimmer 30B22.0%2.6%
Muse Spark40.7%11.3%
Muse Spark 1.146.2%15.1%
Muse Spark 1.245.5%17.7%
Muse Spark 1.348.7%24.9%
Muse Spark Open43.9%14.9%
Muse Wonder50.1%28.3%
Muse Wonder Full53.6%32.9%
Nemotron 3.5 Bolt 74B-A9B24.7%4.3%
Nemotron 3.5 Lightning 30B A3B NVFP410.6%0.0%
Nemotron 3.5 Super VL 120B A12B30.7%10.9%
Nemotron 3.5 Super VL 120B A12B NVFP430.7%10.6%
Nemotron 3 Nano 30B4.6%0.0%
Nemotron 3 Nano Omni 30B A3B4.8%0.0%
Nemotron 3 Ultra28.4%3.1%
Nemotron Ultra 253B7.4%0.0%
North Mini Code11.1%0.3%
Nova Pro3.2%0.0%
o17.0%0.3%
o320.1%1.1%
o4-mini16.5%0.6%
Phi-43.8%0.0%
Phi-4-reasoning3.8%0.0%
Quasar 438B18.7%9.4%
Qwen3.5-122B-A10B25.2%0.6%
Qwen3.5-27B23.9%0.9%
Qwen3.5-35B-A3B21.0%0.9%
Qwen3.5 397B29.0%1.7%
Qwen3.5-4B9.9%0.0%
Qwen3.6-27B23.1%1.1%
Qwen3.6-35B-A3B22.2%0.3%
Qwen 3.6 Max (preview)30.8%3.7%
Qwen3.6 Plus27.8%2.9%
Qwen3.7 Max40.5%13.4%
Qwen3.7 Plus35.6%9.1%
Qwen3.8-27B33.9%5.4%
Qwen3.8-Flash-Next38.0%11.1%
Qwen3.8 Max Preview43.1%17.7%
Qwen3 Max11.9%0.0%
Qwen3-Omni-30B-A3B-Instruct4.6%0.0%
Qwen3-Omni-30B-A3B-Thinking7.5%0.0%
Beam 1.0 501B-A23B34.7%16.3%
Sarvam 105B11.0%0.0%
Sarvam 30B7.5%0.3%
Solar Pro 23.7%0.0%
Solar Pro 310.3%0.0%
Solar Pro 429.8%6.3%
Step 3.7 Flash21.4%2.3%
Step 5 Preview46.5%20.9%
Trinity-Large-Preview15.8%0.9%
Trinity-Large-Thinking15.8%0.9%
Ultravox v0.6 Llama 3.3 70B3.6%0.0%