benchgap
Calibration

AA-Omniscience Accuracy → MMLU

MMLU is estimated from AA-Omniscience Accuracy with a offset logistic curve fitted on 10 models measured on both: y = 0.8033 + (0.9047 − 0.8033) / (1 + exp(−146.94·(x − 0.1970))), R² = 0.98, cross-validated error 0.9 pp. It is used for 144 estimates.

Estimated modelAA-Omniscience AccuracyMMLUSource
Claude Opus 4.5 Thinking46.6%90.5%estimated ± 0.9 pp, medium confidence
Claude Opus 4.645.8%90.5%estimated ± 0.9 pp, medium confidence
Claude Opus 4.6 (Adaptive)47.0%90.5%estimated ± 0.9 pp, medium confidence
Claude Opus 4.748.9%90.5%estimated ± 0.9 pp, medium confidence
Claude Opus 4.848.8%90.5%estimated ± 0.9 pp, medium confidence
Claude Sonnet 5.551.5%90.5%estimated ± 0.9 pp, medium confidence
DeepSeek V4.1 Flash 101746.4%90.5%estimated ± 0.9 pp, medium confidence
DeepSeek V4 Pro49.1%90.5%estimated ± 0.9 pp, medium confidence
Gemini 3.5 Flash51.9%90.5%estimated ± 0.9 pp, medium confidence
Gemini 3.6 Flash50.0%90.5%estimated ± 0.9 pp, medium confidence
Gemini 3.7 Flash55.3%90.5%estimated ± 0.9 pp, medium confidence
Gemini 3.8 Flash50.5%90.5%estimated ± 0.9 pp, medium confidence
Gemini 3 Flash45.8%90.5%estimated ± 0.9 pp, medium confidence
Gemini 4 Argon48.3%90.5%estimated ± 0.9 pp, medium confidence
Gemini 4 Hydrogen50.5%90.5%estimated ± 0.9 pp, medium confidence
GPT-5.244.3%90.5%estimated ± 0.9 pp, medium confidence
GPT-5.450.8%90.5%estimated ± 0.9 pp, medium confidence
GPT-5.6 Terra46.8%90.5%estimated ± 0.9 pp, medium confidence
GPT-6.1 Terra47.6%90.5%estimated ± 0.9 pp, medium confidence
GPT-6 Sol54.5%90.5%estimated ± 0.9 pp, medium confidence
Grok 4.551.6%90.5%estimated ± 0.9 pp, medium confidence
Grok 4.648.2%90.5%estimated ± 0.9 pp, medium confidence
Kimi K3.1 Pro48.9%90.5%estimated ± 0.9 pp, medium confidence
Kimi K347.6%90.5%estimated ± 0.9 pp, medium confidence
MiniMax M344.9%90.5%estimated ± 0.9 pp, medium confidence
MiniMax M3.1 Flash45.1%90.5%estimated ± 0.9 pp, medium confidence
Muse Spark49.6%90.5%estimated ± 0.9 pp, medium confidence
Muse Spark 1.152.1%90.5%estimated ± 0.9 pp, medium confidence
Muse Spark 1.245.4%90.5%estimated ± 0.9 pp, medium confidence
Qwen4 Max43.7%90.5%estimated ± 0.9 pp, medium confidence
Muse Spark 1.343.6%90.5%estimated ± 0.9 pp, medium confidence
GPT-5.6 Luna42.7%90.5%estimated ± 0.9 pp, medium confidence
Grok 4.742.7%90.5%estimated ± 0.9 pp, medium confidence
Muse Spark Open42.0%90.5%estimated ± 0.9 pp, medium confidence
GPT-5.2-Codex41.1%90.5%estimated ± 0.9 pp, medium confidence
Muse Wonder Full41.0%90.5%estimated ± 0.9 pp, medium confidence
Claude Opus 4.540.9%90.5%estimated ± 0.9 pp, medium confidence
Grok 440.5%90.5%estimated ± 0.9 pp, medium confidence
DeepSeek V4 Flash 073140.4%90.5%estimated ± 0.9 pp, medium confidence
GPT-540.3%90.5%estimated ± 0.9 pp, medium confidence
Claude Sonnet 540.1%90.5%estimated ± 0.9 pp, medium confidence
GPT-5.1-Codex39.9%90.5%estimated ± 0.9 pp, medium confidence
GPT-5.1-Codex-Max39.9%90.5%estimated ± 0.9 pp, medium confidence
GPT-6 Luna39.6%90.5%estimated ± 0.9 pp, medium confidence
Inkling39.3%90.5%estimated ± 0.9 pp, medium confidence
Gemini 2.5 Pro39.1%90.5%estimated ± 0.9 pp, medium confidence
Claude Sonnet 4.638.6%90.5%estimated ± 0.9 pp, medium confidence
Muse Wonder38.0%90.5%estimated ± 0.9 pp, medium confidence
Ling 3.0 Tiny37.9%90.5%estimated ± 0.9 pp, medium confidence
Qwen 3.6 Max (preview)37.9%90.5%estimated ± 0.9 pp, medium confidence
GPT-5.137.7%90.5%estimated ± 0.9 pp, medium confidence
GPT-5.4 mini37.5%90.5%estimated ± 0.9 pp, medium confidence
Step 5 Preview36.4%90.5%estimated ± 0.9 pp, medium confidence
Kimi K2.535.2%90.5%estimated ± 0.9 pp, medium confidence
MiMo-V2.6-Pro34.8%90.5%estimated ± 0.9 pp, medium confidence
Claude Haiku 5.534.7%90.5%estimated ± 0.9 pp, medium confidence
Grok 4.334.6%90.5%estimated ± 0.9 pp, medium confidence
Inkling 2 Small33.8%90.5%estimated ± 0.9 pp, high confidence
Kimi K2.632.6%90.5%estimated ± 0.9 pp, high confidence
GLM-5.332.3%90.5%estimated ± 0.9 pp, high confidence
Inkling-Small31.8%90.5%estimated ± 0.9 pp, high confidence
Hy3 Preview31.5%90.5%estimated ± 0.9 pp, high confidence
Qwen3.7 Max31.1%90.5%estimated ± 0.9 pp, high confidence
Beam 1.0 501B-A23B30.7%90.5%estimated ± 0.9 pp, high confidence
Qwen3.8 Max Preview29.5%90.5%estimated ± 0.9 pp, high confidence
GLM-4.729.3%90.5%estimated ± 0.9 pp, high confidence
GLM-5V-Turbo29.3%90.5%estimated ± 0.9 pp, high confidence
GLM-5-Turbo28.4%90.5%estimated ± 0.9 pp, high confidence
Apodex 1.128.2%90.5%estimated ± 0.9 pp, high confidence
Apodex 1.1 Mini28.2%90.5%estimated ± 0.9 pp, high confidence
Claude 3.7 Sonnet28.2%90.5%estimated ± 0.9 pp, high confidence
Claude 3.7 Sonnet Thinking28.0%90.5%estimated ± 0.9 pp, high confidence
Muse Glimmer 30B28.0%90.5%estimated ± 0.9 pp, high confidence
MiMo-V2.6-Flash27.2%90.5%estimated ± 0.9 pp, high confidence
MiniMax M2.726.8%90.5%estimated ± 0.9 pp, high confidence
MiMo-V2-Pro26.6%90.5%estimated ± 0.9 pp, high confidence
Gemini 3.5 Flash-Lite26.5%90.5%estimated ± 0.9 pp, high confidence
Qwen3.6 Plus26.4%90.5%estimated ± 0.9 pp, high confidence
GLM-526.3%90.5%estimated ± 0.9 pp, high confidence
Nemotron 3.5 Super VL 120B A12B26.2%90.5%estimated ± 0.9 pp, high confidence
Gemini 2.5 Flash26.1%90.5%estimated ± 0.9 pp, high confidence
Mistral Large 426.0%90.5%estimated ± 0.9 pp, high confidence
Nemotron 3.5 Super VL 120B A12B NVFP425.9%90.5%estimated ± 0.9 pp, high confidence
Step 3.7 Flash25.8%90.5%estimated ± 0.9 pp, high confidence
GPT-5.4 nano25.7%90.5%estimated ± 0.9 pp, high confidence
DeepSeek V325.5%90.5%estimated ± 0.9 pp, high confidence
Grok 4.1 Fast (Reasoning)25.1%90.5%estimated ± 0.9 pp, high confidence
Mistral Large 325.0%90.5%estimated ± 0.9 pp, high confidence
Qwen3 Max24.4%90.5%estimated ± 0.9 pp, high confidence
DeepSeek V3 032424.3%90.5%estimated ± 0.9 pp, high confidence
GLM-5.224.3%90.5%estimated ± 0.9 pp, high confidence
DeepSeek V3.224.0%90.4%estimated ± 0.9 pp, high confidence
GLM-5.123.7%90.4%estimated ± 0.9 pp, high confidence
Gemini 2.0 Flash23.5%90.4%estimated ± 0.9 pp, high confidence
Grok Code Fast 123.5%90.4%estimated ± 0.9 pp, high confidence
Qwen3.8-Flash-Next23.5%90.4%estimated ± 0.9 pp, high confidence
DeepSeek V3.123.1%90.4%estimated ± 0.9 pp, high confidence
Mistral Medium 3.5 128B23.0%90.4%estimated ± 0.9 pp, high confidence
Grok 4 Fast (Reasoning)22.8%90.4%estimated ± 0.9 pp, high confidence
Claude 4 Sonnet22.7%90.3%estimated ± 0.9 pp, high confidence
MiMo-V2.5-Pro22.4%90.3%estimated ± 0.9 pp, high confidence
Mercury 2.522.0%90.1%estimated ± 0.9 pp, high confidence
Nemotron 3.5 Bolt 74B-A9B BF1621.8%90.0%estimated ± 0.9 pp, high confidence
Nemotron 3 Ultra21.6%89.9%estimated ± 0.9 pp, high confidence
GLM-4.621.4%89.7%estimated ± 0.9 pp, high confidence
Mistral Small 421.4%89.7%estimated ± 0.9 pp, high confidence
Muse Ember21.1%89.3%estimated ± 0.9 pp, high confidence
Qwen3.5-27B20.7%88.6%estimated ± 0.9 pp, high confidence
Magistral Medium20.3%87.5%estimated ± 0.9 pp, high confidence
Qwen3.5-35B-A3B20.1%86.9%estimated ± 0.9 pp, high confidence
Gemma 4 31B19.9%86.2%estimated ± 0.9 pp, high confidence
GPT-4o19.9%86.2%estimated ± 0.9 pp, high confidence
Mistral Large 219.9%86.2%estimated ± 0.9 pp, high confidence
Qwen3.6-27B19.6%85.0%estimated ± 0.9 pp, high confidence
MiMo-V2-Omni19.3%84.0%estimated ± 0.9 pp, high confidence
Solar Pro 318.5%81.8%estimated ± 0.9 pp, high confidence
North Mini Code18.4%81.6%estimated ± 0.9 pp, high confidence
Solar Pro 418.4%81.6%estimated ± 0.9 pp, high confidence
Mistral Medium 318.3%81.5%estimated ± 0.9 pp, high confidence
Ling 3.0 Flash18.2%81.3%estimated ± 0.9 pp, high confidence
Magistral Small18.0%81.1%estimated ± 0.9 pp, high confidence
Claude 3 Haiku17.6%80.8%estimated ± 0.9 pp, high confidence
A.X K217.5%80.7%estimated ± 0.9 pp, high confidence
Qwen3.8-27B17.3%80.6%estimated ± 0.9 pp, high confidence
Grok 4.1 Fast17.2%80.6%estimated ± 0.9 pp, high confidence
Nova Pro16.9%80.5%estimated ± 0.9 pp, high confidence
K-Exaone16.4%80.4%estimated ± 0.9 pp, high confidence
GLM-4.5-Air16.3%80.4%estimated ± 0.9 pp, high confidence
Solar Pro 216.1%80.4%estimated ± 0.9 pp, high confidence
Gemma 4 12B15.7%80.4%estimated ± 0.9 pp, high confidence
Ling 2.6 Flash15.6%80.4%estimated ± 0.9 pp, high confidence
MiMo-V2-Flash15.6%80.4%estimated ± 0.9 pp, high confidence
Gemini 2.5 Flash-Lite15.4%80.4%estimated ± 0.9 pp, high confidence
Devstral Small14.7%80.3%estimated ± 0.9 pp, high confidence
Nemotron 3.5 Lightning 30B A3B NVFP414.7%80.3%estimated ± 0.9 pp, high confidence
K-EXAONE 2.014.5%80.3%estimated ± 0.9 pp, high confidence
Gemma 3 27B13.0%80.3%estimated ± 0.9 pp, medium confidence
Granite 4.2 8B11.8%80.3%estimated ± 0.9 pp, medium confidence
Granite 4.2 30B11.2%80.3%estimated ± 0.9 pp, medium confidence
Command A+9.8%80.3%estimated ± 0.9 pp, medium confidence
Gemma 4 E4B9.7%80.3%estimated ± 0.9 pp, medium confidence
LFM2.5-8B-A1B9.4%80.3%estimated ± 0.9 pp, medium confidence
Granite 4.2 3B9.2%80.3%estimated ± 0.9 pp, medium confidence
LFM3-1.7B7.1%80.3%estimated ± 0.9 pp, medium confidence

Fitted on

ModelAA-Omniscience AccuracyMMLU
Claude 3.5 Haiku13.2%80.3%
DeepSeek-R130.5%90.8%
GPT-4.127.8%90.2%
GPT-4.1 mini20.3%87.5%
GPT-4.1 nano13.7%80.1%
GPT-OSS 120B21.7%90.0%
Kimi K227.4%89.5%
Mistral Small 3.115.1%80.6%
o134.5%91.8%
o4-mini24.8%90.0%