benchgap
Calibration

AA Agentic Index → AA ITBench

AA ITBench is estimated from AA Agentic Index with a linear curve fitted on 17 models measured on both: y = 0.3204·x + 0.3283, R² = 0.61, cross-validated error 4.0 pp. It is used for 40 estimates.

Estimated modelAA Agentic IndexAA ITBenchSource
Claude Opus 556.2%50.8%estimated ± 4.0 pp, high confidence
Muse Spark 1.355.7%50.7%estimated ± 4.0 pp, high confidence
Grok 4.653.4%49.9%estimated ± 4.0 pp, high confidence
Claude Fable 551.0%49.2%estimated ± 4.0 pp, high confidence
Qwen3.8 Max Preview49.6%48.7%estimated ± 4.0 pp, high confidence
DeepSeek V4 Pro49.6%48.7%estimated ± 4.0 pp, high confidence
Qwen3.8-27B46.5%47.7%estimated ± 4.0 pp, high confidence
Claude Sonnet 544.3%47.0%estimated ± 4.0 pp, high confidence
Muse Spark 1.244.0%46.9%estimated ± 4.0 pp, high confidence
Grok 4.542.1%46.3%estimated ± 4.0 pp, high confidence
DeepSeek V4 Flash 073141.7%46.2%estimated ± 4.0 pp, high confidence
Gemini 3.7 Flash36.4%44.5%estimated ± 4.0 pp, high confidence
Quasar 438B32.7%43.3%estimated ± 4.0 pp, high confidence
MiniMax M330.8%42.7%estimated ± 4.0 pp, high confidence
Gemini 3.6 Flash30.2%42.5%estimated ± 4.0 pp, high confidence
Muse Spark 1.127.5%41.7%estimated ± 4.0 pp, high confidence
Hy325.6%41.0%estimated ± 4.0 pp, high confidence
Inkling-Small25.0%40.8%estimated ± 4.0 pp, high confidence
Inkling24.3%40.6%estimated ± 4.0 pp, high confidence
Kimi K2.7 Code22.5%40.0%estimated ± 4.0 pp, high confidence
Nemotron 3 Ultra21.7%39.8%estimated ± 4.0 pp, high confidence
Ling 3.0 Flash21.0%39.5%estimated ± 4.0 pp, high confidence
Qwen3.7 Plus19.7%39.1%estimated ± 4.0 pp, high confidence
GPT-5.4 nano17.7%38.5%estimated ± 4.0 pp, high confidence
Gemini 3.5 Flash-Lite15.9%37.9%estimated ± 4.0 pp, high confidence
Muse Glimmer 30B10.5%36.2%estimated ± 4.0 pp, high confidence
Qwen3.5-122B-A10B9.6%35.9%estimated ± 4.0 pp, high confidence
Nemotron 3.5 Lightning 30B A3B NVFP46.1%34.8%estimated ± 4.0 pp, medium confidence
Granite 4.2 8B3.7%34.0%estimated ± 4.0 pp, medium confidence
Command A+3.6%34.0%estimated ± 4.0 pp, medium confidence
Mistral Large 32.4%33.6%estimated ± 4.0 pp, medium confidence
Mistral Small 41.4%33.3%estimated ± 4.0 pp, medium confidence
GPT-OSS 20B1.4%33.3%estimated ± 4.0 pp, medium confidence
Trinity-Large-Preview1.2%33.2%estimated ± 4.0 pp, medium confidence
Nemotron 3 Nano 30B1.0%33.1%estimated ± 4.0 pp, medium confidence
DeepSeek V30.8%33.1%estimated ± 4.0 pp, medium confidence
Celeris-10.7%33.0%estimated ± 4.0 pp, medium confidence
Llama 4 Maverick0.6%33.0%estimated ± 4.0 pp, medium confidence
Llama 4 Scout0.5%33.0%estimated ± 4.0 pp, medium confidence
Gemma 3 27B0.1%32.9%estimated ± 4.0 pp, medium confidence

Fitted on

ModelAA Agentic IndexAA ITBench
Claude Fable 5.158.0%49.5%
Claude Opus 4.739.5%46.7%
Gemini 3.5 Flash27.3%40.3%
Gemini 3.8 Flash41.1%52.5%
Gemma 4 31B6.7%37.3%
GLM-5.125.2%40.3%
GLM-5.239.4%42.7%
GLM-5.353.4%46.1%
GPT-5.4 mini19.7%35.2%
GPT-5.537.3%45.8%
GPT-5.6 Luna42.7%40.3%
GPT-5.6 Sol50.5%56.2%
GPT-5.6 Terra43.7%51.0%
GPT-6 Astra51.5%48.6%
Kimi K350.6%47.7%
MiMo-V2.5-Pro22.7%38.2%
Qwen3.7 Max23.9%42.5%