Calibration
AA-SciCode → SWE-bench Verified
SWE-bench Verified is estimated from AA-SciCode with a linear curve fitted on 28 models measured on both: y = 1.4978·x + 0.0616, R² = 0.79, cross-validated error 6.6 pp. It is used for 6 estimates.
| Estimated model | AA-SciCode | SWE-bench Verified | Source |
|---|---|---|---|
| DeepSeek V3 0324 | 39.0% | 64.6% | estimated ± 6.6 pp, medium confidence |
| Ling 3.0 Flash VL | 44.2% | 72.4% | estimated ± 6.6 pp, medium confidence |
| Ling 3.0 Tiny | 24.2% | 42.4% | estimated ± 6.6 pp, low confidence |
| Ling 3.1 Flash | 54.1% | 87.2% | estimated ± 6.6 pp, medium confidence |
| North Mini Code | 38.8% | 64.3% | estimated ± 6.6 pp, medium confidence |
| Solar Pro 3 | 25.5% | 44.4% | estimated ± 6.6 pp, low confidence |