Calibration
SciCode → SWE-bench Verified
SWE-bench Verified is estimated from SciCode with a linear curve fitted on 14 models measured on both: y = 1.4035·x + 0.0883, R² = 0.83, cross-validated error 5.7 pp. It is used for 4 estimates.
| Estimated model | SciCode | SWE-bench Verified | Source |
|---|---|---|---|
| A.X K2 | 41.0% | 66.4% | estimated ± 5.7 pp, medium confidence |
| Agents-A1-4B | 29.6% | 50.4% | estimated ± 5.7 pp, medium confidence |
| Granite 4.2 3B | 24.1% | 42.7% | estimated ± 5.7 pp, low confidence |
| Mercury 2.5 | 38.0% | 62.2% | estimated ± 5.7 pp, medium confidence |