Calibration
AA-GPQA Diamond → MMLU-Pro
MMLU-Pro is estimated from AA-GPQA Diamond with a linear curve fitted on 36 models measured on both: y = 0.4381·x + 0.4821, R² = 0.73, cross-validated error 3.9 pp. It is used for 11 estimates.
| Estimated model | AA-GPQA Diamond | MMLU-Pro | Source |
|---|---|---|---|
| Claude 3 Opus | 48.9% | 69.6% | estimated ± 3.9 pp, high confidence |
| Claude 4.1 Opus Thinking | 80.9% | 83.7% | estimated ± 3.9 pp, high confidence |
| DeepSeek R1 Distill Qwen 32B | 61.5% | 75.2% | estimated ± 3.9 pp, high confidence |
| Gemini 1.0 Pro | 27.7% | 60.3% | estimated ± 3.9 pp, medium confidence |
| Gemini 1.5 Pro | 58.9% | 74.0% | estimated ± 3.9 pp, high confidence |
| GLM-5.3-Flash | 91.2% | 88.2% | estimated ± 3.9 pp, high confidence |
| GPT-4o mini | 42.6% | 66.9% | estimated ± 3.9 pp, medium confidence |
| o1-preview | 76.5% | 81.7% | estimated ± 3.9 pp, high confidence |
| o3-pro | 84.5% | 85.2% | estimated ± 3.9 pp, high confidence |
| Phi-4 Multimodal Instruct | 31.5% | 62.0% | estimated ± 3.9 pp, medium confidence |
| Qwen2.5 Coder 32B Instruct | 41.7% | 66.5% | estimated ± 3.9 pp, medium confidence |