Calibration
AA-SciCode → NL2Repo
NL2Repo is estimated from AA-SciCode with a linear curve fitted on 15 models measured on both: y = 1.5999·x + -0.3023, R² = 0.66, cross-validated error 6.5 pp. It is used for 17 estimates.
| Estimated model | AA-SciCode | NL2Repo | Source |
|---|---|---|---|
| Claude Haiku 5.5 | 55.0% | 57.8% | estimated ± 6.5 pp, medium confidence |
| DeepSeek V3 0324 | 39.0% | 32.2% | estimated ± 6.5 pp, medium confidence |
| Gemini 4 Argon | 61.8% | 68.6% | estimated ± 6.5 pp, low confidence |
| GPT-6.1 Sol | 54.2% | 56.5% | estimated ± 6.5 pp, medium confidence |
| GPT-6 Luna | 54.6% | 57.1% | estimated ± 6.5 pp, medium confidence |
| GPT-6 Sol | 57.6% | 61.9% | estimated ± 6.5 pp, medium confidence |
| Grok 4.7 | 57.4% | 61.6% | estimated ± 6.5 pp, medium confidence |
| Ling 3.0 Flash VL | 44.2% | 40.5% | estimated ± 6.5 pp, medium confidence |
| Ling 3.0 Tiny | 24.2% | 8.5% | estimated ± 6.5 pp, low confidence |
| Ling 3.1 Flash | 54.1% | 56.3% | estimated ± 6.5 pp, medium confidence |
| Mercury 2.5 | 38.5% | 31.4% | estimated ± 6.5 pp, medium confidence |
| MiMo-V2.6-Flash | 51.3% | 51.8% | estimated ± 6.5 pp, medium confidence |
| MiMo-V2.6-Pro | 60.9% | 67.2% | estimated ± 6.5 pp, low confidence |
| Mistral Large 4 | 54.2% | 56.5% | estimated ± 6.5 pp, medium confidence |
| North Mini Code | 38.8% | 31.8% | estimated ± 6.5 pp, medium confidence |
| Solar Pro 3 | 25.5% | 10.6% | estimated ± 6.5 pp, low confidence |
| Step 5 Preview | 58.9% | 64.0% | estimated ± 6.5 pp, medium confidence |