Calibration
SimpleQA → AA-GPQA Diamond
AA-GPQA Diamond is estimated from SimpleQA with a linear curve fitted on 9 models measured on both: y = 1.0254·x + 0.4176, R² = 0.52, cross-validated error 11.7 pp. It is used for 0 estimates.
Fitted on
| Model | SimpleQA | AA-GPQA Diamond |
|---|---|---|
| DeepSeek-R1 | 30.1% | 81.3% |
| DeepSeek V3 | 24.9% | 55.7% |
| Gemini 1.5 Flash | 8.6% | 46.3% |
| Gemini 2.0 Flash | 29.9% | 62.3% |
| Gemini 2.5 Flash-Lite | 10.7% | 47.4% |
| Kimi K2 | 31.0% | 76.6% |
| Mistral Small 3.1 | 10.4% | 45.4% |
| o1-mini | 7.6% | 60.3% |
| o4-mini | 20.2% | 78.4% |