benchgap
Calibration

AA-SciCode → FrontierSWE v2

FrontierSWE v2 is estimated from AA-SciCode with a Hill curve fitted on 20 models measured on both: y = 0.0000 + (1.2000 − 0.0000)·x^6.00 / (0.66441^6.00 + x^6.00), R² = 0.49, cross-validated error 14.2 pp. It is used for 33 estimates.

Estimated modelAA-SciCodeFrontierSWE v2Source
Step 5 Preview58.9%39.2%estimated ± 14.2 pp, low confidence
GPT-6.1 Terra55.1%29.5%estimated ± 14.2 pp, low confidence
Qwen3.8 Max Preview52.1%22.6%estimated ± 14.2 pp, low confidence
MiMo-V2.6-Flash51.3%21.0%estimated ± 14.2 pp, low confidence
Hy348.6%15.9%estimated ± 14.2 pp, low confidence
Hy3 Preview48.6%15.9%estimated ± 14.2 pp, low confidence
Quasar 438B48.1%15.1%estimated ± 14.2 pp, low confidence
Solar Pro 448.0%14.9%estimated ± 14.2 pp, low confidence
Gemini 2.5 Pro46.3%12.3%estimated ± 14.2 pp, low confidence
Apodex 1.145.5%11.2%estimated ± 14.2 pp, low confidence
Apodex 1.1 Mini45.5%11.2%estimated ± 14.2 pp, low confidence
Gemma 4 31B45.5%11.2%estimated ± 14.2 pp, low confidence
Ling 3.0 Flash VL44.2%9.6%estimated ± 14.2 pp, low confidence
K-EXAONE 2.042.0%7.2%estimated ± 14.2 pp, low confidence
A.X K241.0%6.3%estimated ± 14.2 pp, low confidence
Trinity-Large-Preview40.6%5.9%estimated ± 14.2 pp, low confidence
Trinity-Large-Thinking40.6%5.9%estimated ± 14.2 pp, low confidence
Gemma 4 26B A4B40.0%5.5%estimated ± 14.2 pp, low confidence
Nemotron 3.5 Bolt 74B-A9B39.7%5.2%estimated ± 14.2 pp, low confidence
Qwen3.5-122B-A10B39.7%5.2%estimated ± 14.2 pp, low confidence
DeepSeek V3 032439.0%4.7%estimated ± 14.2 pp, low confidence
GPT-OSS 20B38.9%4.6%estimated ± 14.2 pp, low confidence
Mistral Small 438.8%4.6%estimated ± 14.2 pp, low confidence
North Mini Code38.8%4.6%estimated ± 14.2 pp, low confidence
Command A+38.5%4.4%estimated ± 14.2 pp, low confidence
Mercury 2.538.5%4.4%estimated ± 14.2 pp, low confidence
Mistral Large 336.6%3.3%estimated ± 14.2 pp, low confidence
DeepSeek V335.8%2.9%estimated ± 14.2 pp, low confidence
GPT-OSS 120B34.0%2.1%estimated ± 14.2 pp, low confidence
Nemotron 3.5 Lightning 30B A3B NVFP432.1%1.5%estimated ± 14.2 pp, low confidence
Llama 4 Maverick31.7%1.4%estimated ± 14.2 pp, low confidence
Mistral Small 3.127.8%0.6%estimated ± 14.2 pp, low confidence
MiniCPM5-2B26.3%0.5%estimated ± 14.2 pp, low confidence

Fitted on

ModelAA-SciCodeFrontierSWE v2
Claude Fable 561.0%47.0%
Claude Fable 5.163.1%56.3%
Claude Fable 5.267.7%69.9%
Claude Haiku 5.555.0%43.8%
Claude Opus 556.4%52.0%
Claude Opus 5.566.9%62.3%
Claude Sonnet 5.561.0%61.9%
Gemini 3.7 Flash57.2%20.3%
Gemini 3.8 Flash56.6%19.6%
Gemini 4 Argon61.8%55.0%
GLM-5.359.0%30.2%
GPT-5.6 Sol57.1%32.2%
GPT-6 Astra56.5%65.5%
Grok 4.656.5%25.3%
Grok 4.757.4%29.5%
Inkling47.0%4.1%
Inkling 2 Small52.3%25.1%
Kimi K359.5%25.9%
Muse Spark 1.257.4%12.0%
Muse Spark 1.358.8%25.8%