Calibration
DeepSWE → SciCode
SciCode is estimated from DeepSWE with a Michaelis–Menten + offset curve fitted on 5 models measured on both: y = 0.1843 + 2.0000·x / (2.70295 + x), R² = 0.64, cross-validated error 7.6 pp. It is used for 14 estimates.
| Estimated model | DeepSWE | SciCode | Source |
|---|---|---|---|
| Qwen4 Max | 75.7% | 62.2% | estimated ± 7.6 pp, low confidence |
| Hy4 | 74.9% | 61.8% | estimated ± 7.6 pp, low confidence |
| Pareto 26.9 | 74.0% | 61.4% | estimated ± 7.6 pp, low confidence |
| MAI Code 2 | 73.2% | 61.0% | estimated ± 7.6 pp, low confidence |
| SWE-2 | 73.0% | 61.0% | estimated ± 7.6 pp, low confidence |
| Particle-1 | 72.6% | 60.8% | estimated ± 7.6 pp, low confidence |
| ERNIE 5.5 Flash Preview | 71.2% | 60.1% | estimated ± 7.6 pp, low confidence |
| Pareto 26.10 Preview | 69.9% | 59.5% | estimated ± 7.6 pp, low confidence |
| Hy4 flash | 64.6% | 57.0% | estimated ± 7.6 pp, low confidence |
| LongCat-2.5 | 64.6% | 57.0% | estimated ± 7.6 pp, low confidence |
| Ultramarine 1 Flash | 61.5% | 55.5% | estimated ± 7.6 pp, low confidence |
| SWE-2 Flash | 60.1% | 54.8% | estimated ± 7.6 pp, low confidence |
| MAI Code 2 Flash | 59.9% | 54.7% | estimated ± 7.6 pp, low confidence |
| Mercury 3 | 46.3% | 47.7% | estimated ± 7.6 pp, low confidence |
Fitted on
| Model | DeepSWE | SciCode |
|---|---|---|
| Beam 1.0 501B-A23B | 43.3% | 49.2% |
| Beam 1.0 501B-A23B FP8 | 43.0% | 48.3% |
| Beam 1.0 501B-A23B NVFP4 | 42.7% | 46.9% |
| Ripple-65B Preview | 44.3% | 39.4% |
| Step 5 Preview | 67.7% | 58.9% |