Calibration
GPQA → AA-GPQA Diamond
AA-GPQA Diamond is estimated from GPQA with a linear curve fitted on 60 models measured on both: y = 1.0083·x + -0.0219, R² = 0.89, cross-validated error 5.4 pp. It is used for 12 estimates.
| Estimated model | GPQA | AA-GPQA Diamond | Source |
|---|---|---|---|
| Claude 3.5 Sonnet | 59.4% | 57.7% | estimated ± 5.4 pp, medium confidence |
| Interfaze Beta | 89.9% | 88.5% | estimated ± 5.4 pp, medium confidence |
| LFM2.5-230M | 25.4% | 23.4% | estimated ± 5.4 pp, low confidence |
| LFM2.5-VL-450M | 25.7% | 23.7% | estimated ± 5.4 pp, low confidence |
| MAI-Thinking-1 | 84.2% | 82.7% | estimated ± 5.4 pp, medium confidence |
| Mellum2-12B-A2.5B-Instruct | 40.9% | 39.1% | estimated ± 5.4 pp, low confidence |
| Mellum2-12B-A2.5B-Thinking | 57.6% | 55.9% | estimated ± 5.4 pp, medium confidence |
| o1-pro | 79.0% | 77.5% | estimated ± 5.4 pp, medium confidence |
| Soofi S 30B-A3B | 43.4% | 41.6% | estimated ± 5.4 pp, medium confidence |
| Ternary Bonsai 2 27B | 85.8% | 84.3% | estimated ± 5.4 pp, medium confidence |
| ZAYA1-74B-Preview | 57.3% | 55.6% | estimated ± 5.4 pp, medium confidence |
| ZAYA1-8B | 71.0% | 69.4% | estimated ± 5.4 pp, medium confidence |