Calibration
SWE-bench Pro → SciCode
SciCode is estimated from SWE-bench Pro with a linear curve fitted on 19 models measured on both: y = 0.4180·x + 0.2421, R² = 0.61, cross-validated error 5.6 pp. It is used for 3 estimates.
| Estimated model | SWE-bench Pro | SciCode | Source |
|---|---|---|---|
| Hy4 preview | 65.7% | 51.7% | estimated ± 5.6 pp, medium confidence |
| Atria Dawn Preview | 59.6% | 49.1% | estimated ± 5.6 pp, medium confidence |
| Laguna S 2.1 | 59.4% | 49.0% | estimated ± 5.6 pp, medium confidence |
Fitted on
| Model | SWE-bench Pro | SciCode |
|---|---|---|
| Gemini 3.5 Flash | 55.1% | 53.1% |
| Granite 4.2 30B | 33.3% | 38.8% |
| Granite 4.2 8B | 19.1% | 36.1% |
| Inkling-Small | 55.9% | 48.7% |
| Kimi K2.6 | 58.6% | 52.2% |
| Kimi K2.5 | 50.7% | 48.7% |
| Ling 3.0 Flash | 56.6% | 41.2% |
| MiniCPM5-2B | 14.4% | 26.3% |
| Muse Glimmer 30B | 51.2% | 43.6% |
| Qwen 3.6 Max (preview) | 57.3% | 47.0% |
| Qwen3.7 Max | 60.6% | 53.5% |
| Qwen3.7 Plus | 57.6% | 51.3% |
| Beam 1.0 501B-A23B | 66.1% | 49.2% |
| Beam 1.0 501B-A23B FP8 | 65.9% | 48.3% |
| Beam 1.0 501B-A23B NVFP4 | 65.6% | 46.9% |
| Sage-A1.5 | 61.2% | 44.3% |
| Sakana Fugu | 59.0% | 60.1% |
| Sakana Fugu-Ultra | 73.7% | 58.7% |
| Spark-X2.6-4B | 54.3% | 36.8% |