benchgap
Calibration

AA-SciCode → FrontierCode 1.1 Main

FrontierCode 1.1 Main is estimated from AA-SciCode with a offset logistic curve fitted on 11 models measured on both: y = 0.0000 + (0.5079 − 0.0000) / (1 + exp(−75.55·(x − 0.5191))), R² = 0.76, cross-validated error 6.6 pp. It is used for 12 estimates.

Estimated modelAA-SciCodeFrontierCode 1.1 MainSource
GPT-6.1 Terra55.1%46.6%estimated ± 6.6 pp, medium confidence
Ling 3.0 Flash VL44.2%0.1%estimated ± 6.6 pp, low confidence
A.X K241.0%0.0%estimated ± 6.6 pp, low confidence
Trinity-Large-Preview40.6%0.0%estimated ± 6.6 pp, low confidence
Trinity-Large-Thinking40.6%0.0%estimated ± 6.6 pp, low confidence
Nemotron 3.5 Bolt 74B-A9B39.7%0.0%estimated ± 6.6 pp, low confidence
DeepSeek V3 032439.0%0.0%estimated ± 6.6 pp, low confidence
GPT-OSS 20B38.9%0.0%estimated ± 6.6 pp, low confidence
Mistral Small 438.8%0.0%estimated ± 6.6 pp, low confidence
North Mini Code38.8%0.0%estimated ± 6.6 pp, low confidence
Command A+38.5%0.0%estimated ± 6.6 pp, low confidence
Mercury 2.538.5%0.0%estimated ± 6.6 pp, low confidence

Fitted on

ModelAA-SciCodeFrontierCode 1.1 Main
Claude Fable 561.0%53.5%
Claude Haiku 5.555.0%46.4%
Claude Opus 4.854.4%46.5%
Claude Opus 556.4%53.4%
Claude Opus 5.566.9%54.4%
Claude Sonnet 554.3%42.7%
Claude Sonnet 5.561.0%46.2%
Gemini 3.7 Flash57.2%43.6%
GPT-5.4 mini52.1%27.0%
GPT-5.555.8%43.0%
GPT-6 Astra56.5%53.3%