benchgap
Calibration

MMLU → AA-Omniscience Accuracy

AA-Omniscience Accuracy is estimated from MMLU with a offset logistic curve fitted on 10 models measured on both: y = 0.1345 + (1.2000 − 0.1345) / (1 + exp(−32.18·(x − 0.9621))), R² = 0.93, cross-validated error 2.4 pp. It is used for 20 estimates.

Estimated modelMMLUAA-Omniscience AccuracySource
GPT-4.590.8%29.3%estimated ± 2.4 pp, high confidence
Sarvam 105B90.6%28.5%estimated ± 2.4 pp, high confidence
Ripple-65B Preview89.6%24.8%estimated ± 2.4 pp, high confidence
Grok 287.5%19.5%estimated ± 2.4 pp, high confidence
Trinity-Large-Preview87.2%19.0%estimated ± 2.4 pp, high confidence
o3-mini86.9%18.5%estimated ± 2.4 pp, high confidence
Grok 2 mini86.2%17.5%estimated ± 2.4 pp, high confidence
Particle-186.1%17.4%estimated ± 2.4 pp, high confidence
Llama 3.1 70B86.0%17.3%estimated ± 2.4 pp, high confidence
Llama 3.3 70B86.0%17.3%estimated ± 2.4 pp, high confidence
LongCat-Flash-Lite-Sparse85.3%16.5%estimated ± 2.4 pp, high confidence
GPT-OSS 20B85.3%16.5%estimated ± 2.4 pp, high confidence
o1-mini85.2%16.4%estimated ± 2.4 pp, high confidence
Sarvam 30B85.1%16.4%estimated ± 2.4 pp, high confidence
Phi-484.8%16.1%estimated ± 2.4 pp, high confidence
GPT-4o mini82.0%14.5%estimated ± 2.4 pp, high confidence
Mixtral 8x22B Instruct v0.177.8%13.7%estimated ± 2.4 pp, medium confidence
Gemma 2 27B76.2%13.6%estimated ± 2.4 pp, medium confidence
Llama 3.1 8B73.0%13.5%estimated ± 2.4 pp, medium confidence
Phi-3.5-mini69.0%13.5%estimated ± 2.4 pp, medium confidence

Fitted on

ModelMMLUAA-Omniscience Accuracy
Claude 3.5 Haiku80.3%13.2%
DeepSeek-R190.8%30.5%
GPT-4.190.2%27.8%
GPT-4.1 mini87.5%20.3%
GPT-4.1 nano80.1%13.7%
GPT-OSS 120B90.0%21.7%
Kimi K289.5%27.4%
Mistral Small 3.180.6%15.1%
o191.8%34.5%
o4-mini90.0%24.8%