Calibration
AA-SciCode → SWE-bench Pro
SWE-bench Pro is estimated from AA-SciCode with a linear curve fitted on 37 models measured on both: y = 1.5480·x + -0.1818, R² = 0.81, cross-validated error 6.8 pp. It is used for 15 estimates.
| Estimated model | AA-SciCode | SWE-bench Pro | Source |
|---|---|---|---|
| Claude Haiku 5.5 | 55.0% | 67.0% | estimated ± 6.8 pp, medium confidence |
| DeepSeek V3 0324 | 39.0% | 42.2% | estimated ± 6.8 pp, medium confidence |
| GPT-6.1 Sol | 54.2% | 65.7% | estimated ± 6.8 pp, medium confidence |
| GPT-6 Luna | 54.6% | 66.3% | estimated ± 6.8 pp, medium confidence |
| GPT-6 Sol | 57.6% | 71.0% | estimated ± 6.8 pp, medium confidence |
| Ling 3.0 Flash VL | 44.2% | 50.2% | estimated ± 6.8 pp, medium confidence |
| Ling 3.0 Tiny | 24.2% | 19.3% | estimated ± 6.8 pp, low confidence |
| Ling 3.1 Flash | 54.1% | 65.6% | estimated ± 6.8 pp, medium confidence |
| Mercury 2.5 | 38.5% | 41.4% | estimated ± 6.8 pp, medium confidence |
| MiMo-V2.6-Flash | 51.3% | 61.2% | estimated ± 6.8 pp, medium confidence |
| MiMo-V2.6-Pro | 60.9% | 76.1% | estimated ± 6.8 pp, medium confidence |
| Mistral Large 4 | 54.2% | 65.7% | estimated ± 6.8 pp, medium confidence |
| North Mini Code | 38.8% | 41.9% | estimated ± 6.8 pp, medium confidence |
| Solar Pro 3 | 25.5% | 21.3% | estimated ± 6.8 pp, low confidence |
| Step 5 Preview | 58.9% | 73.0% | estimated ± 6.8 pp, medium confidence |