Calibration
NL2Repo → Vals SWE-bench
Vals SWE-bench is estimated from NL2Repo with a Michaelis–Menten curve fitted on 13 models measured on both: y = 2.0000·x / (0.69040 + x), R² = 0.72, cross-validated error 5.4 pp. It is used for 15 estimates.
| Estimated model | NL2Repo | Vals SWE-bench | Source |
|---|---|---|---|
| Claude Opus 4.5 | 43.2% | 77.0% | estimated ± 5.4 pp, medium confidence |
| DeepSeek V4.1 Flash | 65.4% | 97.3% | estimated ± 5.4 pp, low confidence |
| dots3-note Preview | 49.8% | 83.8% | estimated ± 5.4 pp, medium confidence |
| Hy4 preview | 58.9% | 92.1% | estimated ± 5.4 pp, medium confidence |
| Ornith-1.0-35B | 34.6% | 66.8% | estimated ± 5.4 pp, low confidence |
| Ornith-1.0-397B | 48.2% | 82.2% | estimated ± 5.4 pp, medium confidence |
| Ornith-1.0-9B | 27.2% | 56.5% | estimated ± 5.4 pp, low confidence |
| Ornith-1.5-35B-A3B | 46.2% | 80.2% | estimated ± 5.4 pp, medium confidence |
| Ornith-1.5-397B | 59.5% | 92.6% | estimated ± 5.4 pp, medium confidence |
| Ornith-1.5-9B | 32.4% | 63.9% | estimated ± 5.4 pp, low confidence |
| Qwen3.6-35B-A3B | 29.4% | 59.7% | estimated ± 5.4 pp, low confidence |
| Qwen3.8-Flash-Next | 48.1% | 82.1% | estimated ± 5.4 pp, medium confidence |
| Qwen3.8-Omni-Flash | 48.9% | 82.9% | estimated ± 5.4 pp, medium confidence |
| Seed 2.1 Pro | 47.0% | 81.0% | estimated ± 5.4 pp, medium confidence |
| Seed 2.1 Turbo | 43.7% | 77.5% | estimated ± 5.4 pp, medium confidence |