Calibration
HLE w/ tools → AA ITBench
AA ITBench is estimated from HLE w/ tools with a linear curve fitted on 5 models measured on both: y = -0.8762·x + 0.9993, R² = 0.82, cross-validated error 3.4 pp. It is used for 17 estimates.
| Estimated model | HLE w/ tools | AA ITBench | Source |
|---|---|---|---|
| Agents-A1 | 47.6% | 58.2% | estimated ± 3.4 pp, low confidence |
| Apodex 1.1 | 56.1% | 50.8% | estimated ± 3.4 pp, medium confidence |
| Claude Haiku 5.5 | 57.4% | 49.6% | estimated ± 3.4 pp, medium confidence |
| Claude Opus 5 | 64.7% | 43.2% | estimated ± 3.4 pp, medium confidence |
| Claude Sonnet 5 | 57.4% | 49.6% | estimated ± 3.4 pp, medium confidence |
| Claude Sonnet 5.5 | 64.5% | 43.4% | estimated ± 3.4 pp, medium confidence |
| DeepSeek V4 Flash 0731 | 45.1% | 60.4% | estimated ± 3.4 pp, low confidence |
| DeepSeek V4 Pro 0813 | 60.0% | 47.4% | estimated ± 3.4 pp, medium confidence |
| dots3-note Preview | 52.6% | 53.8% | estimated ± 3.4 pp, low confidence |
| Hy4 preview | 55.4% | 51.4% | estimated ± 3.4 pp, medium confidence |
| Nemotron 3 Ultra | 37.4% | 67.2% | estimated ± 3.4 pp, low confidence |
| Ornith-1.5-35B-A3B | 33.4% | 70.7% | estimated ± 3.4 pp, low confidence |
| Ornith-1.5-397B | 56.1% | 50.8% | estimated ± 3.4 pp, medium confidence |
| Ornith-1.5-9B | 30.5% | 73.2% | estimated ± 3.4 pp, low confidence |
| Qwen3.7 Max | 53.5% | 53.1% | estimated ± 3.4 pp, low confidence |
| Qwen3.8 Max | 56.2% | 50.7% | estimated ± 3.4 pp, medium confidence |
| Step 3.7 Flash | 47.2% | 58.6% | estimated ± 3.4 pp, low confidence |