Calibration
MMLU → Vals GPQA Diamond
Vals GPQA Diamond is estimated from MMLU with a Hill curve fitted on 8 models measured on both: y = 0.0000 + (1.2000 − 0.0000)·x^6.00 / (0.85404^6.00 + x^6.00), R² = 0.73, cross-validated error 8.2 pp. It is used for 3 estimates.
| Estimated model | MMLU | Vals GPQA Diamond | Source |
|---|---|---|---|
| LongCat-Flash-Lite-Sparse | 85.3% | 59.8% | estimated ± 8.2 pp, medium confidence |
| Mixtral 8x22B Instruct v0.1 | 77.8% | 43.6% | estimated ± 8.2 pp, low confidence |
| Gemma 2 27B | 76.2% | 40.2% | estimated ± 8.2 pp, low confidence |
Fitted on
| Model | MMLU | Vals GPQA Diamond |
|---|---|---|
| Claude 3.5 Haiku | 80.3% | 37.9% |
| GPT-4.1 | 90.2% | 65.4% |
| GPT-4.1 mini | 87.5% | 67.9% |
| GPT-4.1 nano | 80.1% | 50.8% |
| GPT-4o mini | 82.0% | 44.2% |
| o1 | 91.8% | 73.2% |
| o3-mini | 86.9% | 75.5% |
| o4-mini | 90.0% | 74.5% |