Calibration
AA-HLE → ARC-AGI-3
ARC-AGI-3 is estimated from AA-HLE with a offset logistic curve fitted on 16 models measured on both: y = 0.0157 + (0.4862 − 0.0157) / (1 + exp(−200.00·(x − 0.5041))), R² = 0.89, cross-validated error 10.7 pp. It is used for 10 estimates.
| Estimated model | AA-HLE | ARC-AGI-3 | Source |
|---|---|---|---|
| Claude 3 Opus | 2.8% | 1.6% | estimated ± 10.7 pp, low confidence |
| Claude 4.1 Opus Thinking | 12.5% | 1.6% | estimated ± 10.7 pp, low confidence |
| Claude Haiku 5.5 | 44.4% | 1.6% | estimated ± 10.7 pp, low confidence |
| DeepSeek R1 Distill Qwen 32B | 4.6% | 1.6% | estimated ± 10.7 pp, low confidence |
| Gemini 1.0 Pro | 4.2% | 1.6% | estimated ± 10.7 pp, low confidence |
| Gemini 1.5 Pro | 4.6% | 1.6% | estimated ± 10.7 pp, low confidence |
| GPT-4 Turbo | 3.1% | 1.6% | estimated ± 10.7 pp, low confidence |
| GPT-4o mini | 4.2% | 1.6% | estimated ± 10.7 pp, low confidence |
| Phi-4 Multimodal Instruct | 5.0% | 1.6% | estimated ± 10.7 pp, low confidence |
| Qwen2.5 Coder 32B Instruct | 3.5% | 1.6% | estimated ± 10.7 pp, low confidence |