Calibration
MMLU → AA-Omniscience Accuracy
AA-Omniscience Accuracy is estimated from MMLU with a offset logistic curve fitted on 10 models measured on both: y = 0.1345 + (1.2000 − 0.1345) / (1 + exp(−32.18·(x − 0.9621))), R² = 0.93, cross-validated error 2.4 pp. It is used for 20 estimates.
| Estimated model | MMLU | AA-Omniscience Accuracy | Source |
|---|---|---|---|
| GPT-4.5 | 90.8% | 29.3% | estimated ± 2.4 pp, high confidence |
| Sarvam 105B | 90.6% | 28.5% | estimated ± 2.4 pp, high confidence |
| Ripple-65B Preview | 89.6% | 24.8% | estimated ± 2.4 pp, high confidence |
| Grok 2 | 87.5% | 19.5% | estimated ± 2.4 pp, high confidence |
| Trinity-Large-Preview | 87.2% | 19.0% | estimated ± 2.4 pp, high confidence |
| o3-mini | 86.9% | 18.5% | estimated ± 2.4 pp, high confidence |
| Grok 2 mini | 86.2% | 17.5% | estimated ± 2.4 pp, high confidence |
| Particle-1 | 86.1% | 17.4% | estimated ± 2.4 pp, high confidence |
| Llama 3.1 70B | 86.0% | 17.3% | estimated ± 2.4 pp, high confidence |
| Llama 3.3 70B | 86.0% | 17.3% | estimated ± 2.4 pp, high confidence |
| LongCat-Flash-Lite-Sparse | 85.3% | 16.5% | estimated ± 2.4 pp, high confidence |
| GPT-OSS 20B | 85.3% | 16.5% | estimated ± 2.4 pp, high confidence |
| o1-mini | 85.2% | 16.4% | estimated ± 2.4 pp, high confidence |
| Sarvam 30B | 85.1% | 16.4% | estimated ± 2.4 pp, high confidence |
| Phi-4 | 84.8% | 16.1% | estimated ± 2.4 pp, high confidence |
| GPT-4o mini | 82.0% | 14.5% | estimated ± 2.4 pp, high confidence |
| Mixtral 8x22B Instruct v0.1 | 77.8% | 13.7% | estimated ± 2.4 pp, medium confidence |
| Gemma 2 27B | 76.2% | 13.6% | estimated ± 2.4 pp, medium confidence |
| Llama 3.1 8B | 73.0% | 13.5% | estimated ± 2.4 pp, medium confidence |
| Phi-3.5-mini | 69.0% | 13.5% | estimated ± 2.4 pp, medium confidence |
Fitted on
| Model | MMLU | AA-Omniscience Accuracy |
|---|---|---|
| Claude 3.5 Haiku | 80.3% | 13.2% |
| DeepSeek-R1 | 90.8% | 30.5% |
| GPT-4.1 | 90.2% | 27.8% |
| GPT-4.1 mini | 87.5% | 20.3% |
| GPT-4.1 nano | 80.1% | 13.7% |
| GPT-OSS 120B | 90.0% | 21.7% |
| Kimi K2 | 89.5% | 27.4% |
| Mistral Small 3.1 | 80.6% | 15.1% |
| o1 | 91.8% | 34.5% |
| o4-mini | 90.0% | 24.8% |