Calibration
NL2Repo → DeepSWE
DeepSWE is estimated from NL2Repo with a inverse Michaelis–Menten curve fitted on 12 models measured on both: y = 1.61807·(x − 0.0306) / (0.0306 + 2.0000 − x), R² = 0.55, cross-validated error 9.6 pp. It is used for 8 estimates.
| Estimated model | NL2Repo | DeepSWE | Source |
|---|---|---|---|
| Claude Opus 4.5 | 43.2% | 40.6% | estimated ± 9.6 pp, medium confidence |
| dots3-note Preview | 49.8% | 49.3% | estimated ± 9.6 pp, medium confidence |
| Ornith-1.0-35B | 34.6% | 30.3% | estimated ± 9.6 pp, low confidence |
| Ornith-1.0-397B | 48.2% | 47.2% | estimated ± 9.6 pp, medium confidence |
| Ornith-1.0-9B | 27.2% | 22.2% | estimated ± 9.6 pp, low confidence |
| Ornith-1.5-9B | 32.4% | 27.8% | estimated ± 9.6 pp, low confidence |
| Seed 2.1 Pro | 47.0% | 45.6% | estimated ± 9.6 pp, medium confidence |
| Seed 2.1 Turbo | 43.7% | 41.3% | estimated ± 9.6 pp, medium confidence |