benchgap
Calibration

HLE → AA-Omniscience Accuracy

AA-Omniscience Accuracy is estimated from HLE with a Hill curve fitted on 53 models measured on both: y = 0.2083 + (1.2000 − 0.2083)·x^2.77 / (0.79711^2.77 + x^2.77), R² = 0.63, cross-validated error 9.0 pp. It is used for 4 estimates.

Estimated modelHLEAA-Omniscience AccuracySource
Hy459.1%50.9%estimated ± 9.0 pp, medium confidence
Hy4 flash55.2%47.1%estimated ± 9.0 pp, medium confidence
Agents-A147.6%40.0%estimated ± 9.0 pp, medium confidence
Ling 3.1 Flash37.4%31.7%estimated ± 9.0 pp, medium confidence

Fitted on

ModelHLEAA-Omniscience Accuracy
A.X K227.8%17.5%
Apodex 1.156.1%28.2%
Claude Fable 5.165.0%68.4%
Claude Fable 5.274.2%79.1%
Claude Opus 4.530.8%40.9%
Claude Opus 4.653.0%45.8%
Claude Opus 4.754.7%48.9%
Claude Opus 4.857.9%48.8%
Claude Opus 564.7%60.9%
Claude Sonnet 4.649.0%38.6%
Claude Sonnet 557.4%40.1%
DeepSeek V4.1 Flash 101742.6%46.4%
DeepSeek V4 Flash 073134.8%40.4%
DeepSeek V4 Pro42.7%49.1%
Gemini 2.5 Pro18.8%39.1%
Gemini 3.5 Flash40.2%51.9%
Gemma 4 31B26.5%19.9%
GLM-4.724.8%29.3%
GLM-530.5%26.3%
GLM-5.131.0%23.7%
GLM-5.240.5%24.3%
GPT-5.452.1%50.8%
GPT-5.4 mini41.5%37.5%
GPT-5.4 nano37.7%25.7%
GPT-5.552.2%58.0%
GPT-OSS 120B14.9%21.7%
Grok 4.335.0%34.6%
Hy3 Preview25.5%31.5%
Inkling46.0%39.3%
Inkling 2 Small53.2%33.8%
Inkling-Small47.8%31.8%
K-EXAONE 2.018.3%14.5%
Kimi K2.634.7%32.6%
Kimi K3.1 Pro70.0%48.9%
Kimi K24.7%27.4%
Kimi K2.530.1%35.2%
Kimi K356.0%47.6%
LFM3-1.7B7.2%7.1%
Ling 3.0 Flash22.7%18.2%
MiMo-V2.5-Pro48.0%22.4%
MiniMax M2.728.0%26.8%
Muse Spark50.4%49.6%
Muse Spark 1.162.1%52.1%
Nemotron 3.5 Super VL 120B A12B29.2%26.2%
Nemotron 3.5 Super VL 120B A12B NVFP429.0%25.9%
Nemotron 3 Ultra26.7%21.6%
Qwen3.6-27B24.0%19.6%
Qwen3.6 Plus28.8%26.4%
Qwen3.7 Max41.4%31.1%
Qwen3.8-27B30.8%17.3%
Qwen3.8-Flash-Next35.9%23.5%
Qwen4 Max51.7%43.7%
Step 5 Preview46.5%36.4%