Calibration
AA Agentic Index → AA ITBench
AA ITBench is estimated from AA Agentic Index with a linear curve fitted on 17 models measured on both: y = 0.3204·x + 0.3283, R² = 0.61, cross-validated error 4.0 pp. It is used for 40 estimates.
| Estimated model | AA Agentic Index | AA ITBench | Source |
|---|---|---|---|
| Claude Opus 5 | 56.2% | 50.8% | estimated ± 4.0 pp, high confidence |
| Muse Spark 1.3 | 55.7% | 50.7% | estimated ± 4.0 pp, high confidence |
| Grok 4.6 | 53.4% | 49.9% | estimated ± 4.0 pp, high confidence |
| Claude Fable 5 | 51.0% | 49.2% | estimated ± 4.0 pp, high confidence |
| Qwen3.8 Max Preview | 49.6% | 48.7% | estimated ± 4.0 pp, high confidence |
| DeepSeek V4 Pro | 49.6% | 48.7% | estimated ± 4.0 pp, high confidence |
| Qwen3.8-27B | 46.5% | 47.7% | estimated ± 4.0 pp, high confidence |
| Claude Sonnet 5 | 44.3% | 47.0% | estimated ± 4.0 pp, high confidence |
| Muse Spark 1.2 | 44.0% | 46.9% | estimated ± 4.0 pp, high confidence |
| Grok 4.5 | 42.1% | 46.3% | estimated ± 4.0 pp, high confidence |
| DeepSeek V4 Flash 0731 | 41.7% | 46.2% | estimated ± 4.0 pp, high confidence |
| Gemini 3.7 Flash | 36.4% | 44.5% | estimated ± 4.0 pp, high confidence |
| Quasar 438B | 32.7% | 43.3% | estimated ± 4.0 pp, high confidence |
| MiniMax M3 | 30.8% | 42.7% | estimated ± 4.0 pp, high confidence |
| Gemini 3.6 Flash | 30.2% | 42.5% | estimated ± 4.0 pp, high confidence |
| Muse Spark 1.1 | 27.5% | 41.7% | estimated ± 4.0 pp, high confidence |
| Hy3 | 25.6% | 41.0% | estimated ± 4.0 pp, high confidence |
| Inkling-Small | 25.0% | 40.8% | estimated ± 4.0 pp, high confidence |
| Inkling | 24.3% | 40.6% | estimated ± 4.0 pp, high confidence |
| Kimi K2.7 Code | 22.5% | 40.0% | estimated ± 4.0 pp, high confidence |
| Nemotron 3 Ultra | 21.7% | 39.8% | estimated ± 4.0 pp, high confidence |
| Ling 3.0 Flash | 21.0% | 39.5% | estimated ± 4.0 pp, high confidence |
| Qwen3.7 Plus | 19.7% | 39.1% | estimated ± 4.0 pp, high confidence |
| GPT-5.4 nano | 17.7% | 38.5% | estimated ± 4.0 pp, high confidence |
| Gemini 3.5 Flash-Lite | 15.9% | 37.9% | estimated ± 4.0 pp, high confidence |
| Muse Glimmer 30B | 10.5% | 36.2% | estimated ± 4.0 pp, high confidence |
| Qwen3.5-122B-A10B | 9.6% | 35.9% | estimated ± 4.0 pp, high confidence |
| Nemotron 3.5 Lightning 30B A3B NVFP4 | 6.1% | 34.8% | estimated ± 4.0 pp, medium confidence |
| Granite 4.2 8B | 3.7% | 34.0% | estimated ± 4.0 pp, medium confidence |
| Command A+ | 3.6% | 34.0% | estimated ± 4.0 pp, medium confidence |
| Mistral Large 3 | 2.4% | 33.6% | estimated ± 4.0 pp, medium confidence |
| Mistral Small 4 | 1.4% | 33.3% | estimated ± 4.0 pp, medium confidence |
| GPT-OSS 20B | 1.4% | 33.3% | estimated ± 4.0 pp, medium confidence |
| Trinity-Large-Preview | 1.2% | 33.2% | estimated ± 4.0 pp, medium confidence |
| Nemotron 3 Nano 30B | 1.0% | 33.1% | estimated ± 4.0 pp, medium confidence |
| DeepSeek V3 | 0.8% | 33.1% | estimated ± 4.0 pp, medium confidence |
| Celeris-1 | 0.7% | 33.0% | estimated ± 4.0 pp, medium confidence |
| Llama 4 Maverick | 0.6% | 33.0% | estimated ± 4.0 pp, medium confidence |
| Llama 4 Scout | 0.5% | 33.0% | estimated ± 4.0 pp, medium confidence |
| Gemma 3 27B | 0.1% | 32.9% | estimated ± 4.0 pp, medium confidence |
Fitted on
| Model | AA Agentic Index | AA ITBench |
|---|---|---|
| Claude Fable 5.1 | 58.0% | 49.5% |
| Claude Opus 4.7 | 39.5% | 46.7% |
| Gemini 3.5 Flash | 27.3% | 40.3% |
| Gemini 3.8 Flash | 41.1% | 52.5% |
| Gemma 4 31B | 6.7% | 37.3% |
| GLM-5.1 | 25.2% | 40.3% |
| GLM-5.2 | 39.4% | 42.7% |
| GLM-5.3 | 53.4% | 46.1% |
| GPT-5.4 mini | 19.7% | 35.2% |
| GPT-5.5 | 37.3% | 45.8% |
| GPT-5.6 Luna | 42.7% | 40.3% |
| GPT-5.6 Sol | 50.5% | 56.2% |
| GPT-5.6 Terra | 43.7% | 51.0% |
| GPT-6 Astra | 51.5% | 48.6% |
| Kimi K3 | 50.6% | 47.7% |
| MiMo-V2.5-Pro | 22.7% | 38.2% |
| Qwen3.7 Max | 23.9% | 42.5% |