benchgap
Calibration

AA-SciCode → DeepSWE

DeepSWE is estimated from AA-SciCode with a Hill curve fitted on 40 models measured on both: y = 0.0000 + (0.8063 − 0.0000)·x^6.00 / (0.42943^6.00 + x^6.00), R² = 0.39, cross-validated error 7.5 pp. It is used for 33 estimates.

Estimated modelAA-SciCodeDeepSWESource
Solar Pro 448.0%53.3%estimated ± 7.5 pp, low confidence
Nemotron 3.5 Super VL 120B A12B47.8%52.8%estimated ± 7.5 pp, low confidence
Nemotron 3.5 Super VL 120B A12B NVFP447.8%52.8%estimated ± 7.5 pp, low confidence
Gemini 2.5 Pro46.3%49.3%estimated ± 7.5 pp, low confidence
Gemma 4 31B45.5%47.2%estimated ± 7.5 pp, low confidence
Muse Glimmer 30B44.9%45.7%estimated ± 7.5 pp, low confidence
Qwen3.5 397B44.8%45.4%estimated ± 7.5 pp, low confidence
Ling 3.0 Flash VL44.2%43.8%estimated ± 7.5 pp, low confidence
Step 3.7 Flash43.9%43.0%estimated ± 7.5 pp, low confidence
K-EXAONE 2.042.0%37.6%estimated ± 7.5 pp, low confidence
A.X K241.0%34.8%estimated ± 7.5 pp, low confidence
Trinity-Large-Preview40.6%33.6%estimated ± 7.5 pp, low confidence
Trinity-Large-Thinking40.6%33.6%estimated ± 7.5 pp, low confidence
Gemma 4 26B A4B40.0%31.9%estimated ± 7.5 pp, low confidence
Nemotron 3.5 Bolt 74B-A9B39.7%31.0%estimated ± 7.5 pp, low confidence
Qwen3.5-122B-A10B39.7%31.0%estimated ± 7.5 pp, low confidence
DeepSeek V3 032439.0%29.0%estimated ± 7.5 pp, low confidence
GPT-OSS 20B38.9%28.7%estimated ± 7.5 pp, low confidence
Mistral Small 438.8%28.4%estimated ± 7.5 pp, low confidence
North Mini Code38.8%28.4%estimated ± 7.5 pp, low confidence
Command A+38.5%27.6%estimated ± 7.5 pp, low confidence
Mercury 2.538.5%27.6%estimated ± 7.5 pp, low confidence
Granite 4.2 30B37.8%25.6%estimated ± 7.5 pp, low confidence
Mistral Large 336.6%22.3%estimated ± 7.5 pp, low confidence
Qwen3.6-35B-A3B36.6%22.3%estimated ± 7.5 pp, low confidence
DeepSeek V335.8%20.3%estimated ± 7.5 pp, low confidence
GPT-OSS 120B34.0%15.9%estimated ± 7.5 pp, low confidence
Nemotron 3.5 Lightning 30B A3B NVFP432.1%12.0%estimated ± 7.5 pp, low confidence
Llama 4 Maverick31.7%11.2%estimated ± 7.5 pp, low confidence
Granite 4.2 8B31.5%10.9%estimated ± 7.5 pp, low confidence
Mistral Small 3.127.8%5.5%estimated ± 7.5 pp, low confidence
MiniCPM5-2B26.3%4.0%estimated ± 7.5 pp, low confidence
Solar Pro 325.5%3.4%estimated ± 7.5 pp, low confidence

Fitted on

ModelAA-SciCodeDeepSWE
Claude Fable 5.163.1%67.4%
Claude Fable 5.267.7%74.3%
Claude Opus 556.4%68.8%
Claude Opus 5.566.9%74.2%
Claude Sonnet 5.561.0%71.0%
DeepSeek V4.1 Flash 101751.9%77.0%
DeepSeek V4 Flash 073150.3%54.4%
DeepSeek V4 Pro51.0%62.7%
Gemini 3.6 Flash53.4%49.0%
Gemini 3.7 Flash57.2%65.3%
Gemini 3.8 Flash56.6%73.8%
Gemini 4 Argon61.8%77.9%
GLM-5.359.0%66.9%
GPT-5.6 Luna53.6%67.2%
GPT-5.6 Sol57.1%72.7%
GPT-5.6 Terra55.0%69.6%
GPT-6.1 Sol54.2%71.9%
GPT-6 Astra56.5%74.1%
GPT-6 Luna54.6%66.6%
GPT-6 Sol57.6%68.8%
Grok 4.555.0%53.0%
Grok 4.656.5%65.9%
Grok 4.757.4%71.0%
Inkling 2 Small52.3%72.1%
Kimi K3.1 Pro60.8%75.5%
Kimi K359.5%67.5%
Ling 3.1 Flash54.1%59.7%
MiMo-V2.6-Flash51.3%67.9%
MiMo-V2.6-Pro60.9%71.9%
MiniMax M3.1 Flash53.4%73.7%
Muse Spark 1.158.8%53.3%
Muse Spark 1.257.4%59.3%
Muse Spark 1.358.8%75.4%
Muse Spark Open58.4%60.0%
Muse Wonder60.3%78.8%
Muse Wonder Full62.6%79.0%
Qwen3.8-27B46.6%42.2%
Qwen3.8-Flash-Next50.6%58.7%
Beam 1.0 501B-A23B48.5%43.3%
Step 5 Preview58.9%67.7%