Calibration
GPQA → ARC-AGI-2
ARC-AGI-2 is estimated from GPQA with a linear curve fitted on 18 models measured on both: y = 6.4573·x + -5.2816, R² = 0.82, cross-validated error 11.6 pp. It is used for 31 estimates.
| Estimated model | GPQA | ARC-AGI-2 | Source |
|---|---|---|---|
| Claude 3.5 Sonnet | 59.4% | 0.0% | estimated ± 11.6 pp, low confidence |
| Claude Mythos 5 | 94.1% | 79.5% | estimated ± 11.6 pp, low confidence |
| Claude Opus 4.5 | 87.0% | 33.6% | estimated ± 11.6 pp, low confidence |
| DeepSeek V3 | 59.1% | 0.0% | estimated ± 11.6 pp, low confidence |
| Gemini 2.5 Pro | 83.0% | 7.8% | estimated ± 11.6 pp, low confidence |
| Gemma 4 31B | 84.3% | 16.2% | estimated ± 11.6 pp, low confidence |
| Gemma 4 E2B | 43.4% | 0.0% | estimated ± 11.6 pp, low confidence |
| Gemma 4 E4B | 58.6% | 0.0% | estimated ± 11.6 pp, low confidence |
| GLM-4.7 | 85.7% | 25.2% | estimated ± 11.6 pp, low confidence |
| GPT-4.1 | 66.3% | 0.0% | estimated ± 11.6 pp, low confidence |
| GPT-4.1 mini | 64.2% | 0.0% | estimated ± 11.6 pp, low confidence |
| GPT-4.1 nano | 50.3% | 0.0% | estimated ± 11.6 pp, low confidence |
| Granite 4.2 30B | 66.4% | 0.0% | estimated ± 11.6 pp, low confidence |
| Granite 4.2 3B | 54.8% | 0.0% | estimated ± 11.6 pp, low confidence |
| Granite 4.2 8B | 64.1% | 0.0% | estimated ± 11.6 pp, low confidence |
| Grok 4.3 | 90.1% | 53.6% | estimated ± 11.6 pp, low confidence |
| Kimi K2.5 (Reasoning) | 87.6% | 37.5% | estimated ± 11.6 pp, low confidence |
| LFM2.5-VL-450M | 25.7% | 0.0% | estimated ± 11.6 pp, low confidence |
| Ling 2.6 Flash | 59.0% | 0.0% | estimated ± 11.6 pp, low confidence |
| MiMo-V2-Flash | 83.7% | 12.3% | estimated ± 11.6 pp, low confidence |
| o1 | 75.7% | 0.0% | estimated ± 11.6 pp, low confidence |
| o1-pro | 79.0% | 0.0% | estimated ± 11.6 pp, low confidence |
| o3-mini | 77.2% | 0.0% | estimated ± 11.6 pp, low confidence |
| Qwen3 235B 2507 | 77.5% | 0.0% | estimated ± 11.6 pp, low confidence |
| Qwen3.5-122B-A10B | 86.6% | 31.0% | estimated ± 11.6 pp, low confidence |
| Qwen3.5-27B | 85.5% | 23.9% | estimated ± 11.6 pp, low confidence |
| Qwen3.5-35B-A3B | 84.2% | 15.5% | estimated ± 11.6 pp, low confidence |
| Qwen3.5 397B | 88.4% | 42.7% | estimated ± 11.6 pp, low confidence |
| Qwen3.6-27B | 87.8% | 38.8% | estimated ± 11.6 pp, low confidence |
| Qwen3.6-35B-A3B | 86.0% | 27.2% | estimated ± 11.6 pp, low confidence |
| Qwen3.6 Plus | 90.4% | 55.6% | estimated ± 11.6 pp, low confidence |