Calibration
Vals SWE-bench → SciCode
SciCode is estimated from Vals SWE-bench with a Michaelis–Menten curve fitted on 8 models measured on both: y = 1.2863·x / (1.20621 + x), R² = 0.31, cross-validated error 4.5 pp. It is used for 2 estimates.
| Estimated model | Vals SWE-bench | SciCode | Source |
|---|---|---|---|
| Composer 2.5 | 79.6% | 51.1% | estimated ± 4.5 pp, medium confidence |
| Gemini 3.1 Flash-Lite | 62.8% | 44.0% | estimated ± 4.5 pp, low confidence |
Fitted on
| Model | Vals SWE-bench | SciCode |
|---|---|---|
| Gemini 3.5 Flash | 78.8% | 53.1% |
| Grok 4.3 | 71.4% | 47.3% |
| Inkling-Small | 82.2% | 48.7% |
| Kimi K2.6 | 76.2% | 52.2% |
| Ling 3.0 Flash | 65.2% | 41.2% |
| Nemotron 3 Ultra | 69.0% | 44.6% |
| Qwen 3.6 Max (preview) | 72.8% | 47.0% |
| Qwen3.7 Max | 68.8% | 53.5% |