Calibration
SimpleQA → AA-Omniscience Accuracy
AA-Omniscience Accuracy is estimated from SimpleQA with a Michaelis–Menten curve fitted on 7 models measured on both: y = 0.4561·x / (0.19959 + x), R² = 0.86, cross-validated error 2.7 pp. It is used for 4 estimates.
| Estimated model | SimpleQA | AA-Omniscience Accuracy | Source |
|---|---|---|---|
| Grok 3 [Beta] | 43.6% | 31.3% | estimated ± 2.7 pp, low confidence |
| DeepSeek-R1-0528 | 27.8% | 26.5% | estimated ± 2.7 pp, medium confidence |
| Gemini 2.0 Flash-Lite | 16.5% | 20.6% | estimated ± 2.7 pp, medium confidence |
| Gemini 1.5 Flash | 8.6% | 13.7% | estimated ± 2.7 pp, low confidence |
Fitted on
| Model | SimpleQA | AA-Omniscience Accuracy |
|---|---|---|
| DeepSeek-R1 | 30.1% | 30.5% |
| DeepSeek V3 | 24.9% | 25.5% |
| Gemini 2.0 Flash | 29.9% | 23.5% |
| Gemini 2.5 Flash-Lite | 10.7% | 15.4% |
| Kimi K2 | 31.0% | 27.4% |
| Mistral Small 3.1 | 10.4% | 15.1% |
| o4-mini | 20.2% | 24.8% |