benchgap
Calibration

HLE → AA-HLE

AA-HLE is estimated from HLE with a Michaelis–Menten + offset curve fitted on 59 models measured on both: y = 0.0398 + 2.0000·x / (2.15015 + x), R² = 0.69, cross-validated error 6.5 pp. It is used for 3 estimates.

Estimated modelHLEAA-HLESource
Hy459.1%47.1%estimated ± 6.5 pp, medium confidence
Hy4 flash55.2%44.8%estimated ± 6.5 pp, medium confidence
Agents-A147.6%40.2%estimated ± 6.5 pp, medium confidence

Fitted on

ModelHLEAA-HLE
A.X K227.8%29.6%
Apodex 1.156.1%34.1%
Claude Fable 5.165.0%59.1%
Claude Fable 5.274.2%74.3%
Claude Opus 4.530.8%13.2%
Claude Opus 4.653.0%19.1%
Claude Opus 4.754.7%42.3%
Claude Opus 4.857.9%48.7%
Claude Opus 564.7%54.9%
Claude Sonnet 4.649.0%13.3%
Claude Sonnet 557.4%41.3%
DeepSeek V4.1 Flash 101742.6%39.2%
DeepSeek V4 Flash 073134.8%38.6%
DeepSeek V4 Pro42.7%41.0%
Gemini 2.5 Pro18.8%22.5%
Gemini 3.5 Flash40.2%42.7%
Gemma 4 26B A4B17.2%19.3%
Gemma 4 31B26.5%23.6%
GLM-4.724.8%27.4%
GLM-530.5%29.3%
GLM-5.131.0%30.1%
GLM-5.240.5%41.1%
GPT-5.452.1%43.7%
GPT-5.4 mini41.5%28.1%
GPT-5.4 nano37.7%28.3%
GPT-5.552.2%45.8%
GPT-OSS 120B14.9%19.6%
GPT-OSS 20B10.9%11.0%
Grok 4.335.0%37.2%
Hy3 Preview25.5%33.5%
Inkling46.0%31.9%
Inkling 2 Small53.2%44.1%
Inkling-Small47.8%33.3%
K-EXAONE 2.018.3%18.6%
Kimi K2.634.7%37.5%
Kimi K3.1 Pro70.0%52.5%
Kimi K24.7%7.4%
Kimi K2.530.1%30.7%
Kimi K356.0%46.9%
Ling 3.0 Flash22.7%23.7%
Ling 3.1 Flash37.4%39.4%
MiMo-V2.5-Pro48.0%35.7%
MiniCPM5-2B8.9%8.9%
MiniMax M2.728.0%29.6%
Muse Spark50.4%40.7%
Muse Spark 1.162.1%46.2%
Nemotron 3.5 Super VL 120B A12B29.2%30.7%
Nemotron 3.5 Super VL 120B A12B NVFP429.0%30.7%
Nemotron 3 Nano 30B10.6%4.6%
Nemotron 3 Ultra26.7%28.4%
Qwen3.5 397B28.7%29.0%
Qwen3.6-27B24.0%23.1%
Qwen3.6-35B-A3B21.4%22.2%
Qwen3.6 Plus28.8%27.8%
Qwen3.7 Max41.4%40.5%
Qwen3.7 Plus34.7%35.6%
Qwen3.8-27B30.8%33.9%
Qwen3.8-Flash-Next35.9%38.0%
Step 5 Preview46.5%46.5%