Calibration
NL2Repo → AA-SciCode
AA-SciCode is estimated from NL2Repo with a Hill curve fitted on 15 models measured on both: y = 0.2680 + (0.5469 − 0.2680)·x^4.57 / (0.33514^4.57 + x^4.57), R² = 0.78, cross-validated error 3.0 pp. It is used for 14 estimates.
| Estimated model | NL2Repo | AA-SciCode | Source |
|---|---|---|---|
| Claude Opus 4.5 | 43.2% | 48.0% | estimated ± 3.0 pp, high confidence |
| dots3-note Preview | 49.8% | 50.8% | estimated ± 3.0 pp, high confidence |
| Hy4 preview | 58.9% | 52.7% | estimated ± 3.0 pp, high confidence |
| Ornith-1.0-35B | 34.6% | 41.8% | estimated ± 3.0 pp, high confidence |
| Ornith-1.0-397B | 48.2% | 50.2% | estimated ± 3.0 pp, high confidence |
| Ornith-1.0-9B | 27.2% | 34.6% | estimated ± 3.0 pp, medium confidence |
| Ornith-1.5-35B-A3B | 46.2% | 49.5% | estimated ± 3.0 pp, high confidence |
| Ornith-1.5-397B | 59.5% | 52.8% | estimated ± 3.0 pp, high confidence |
| Ornith-1.5-9B | 32.4% | 39.7% | estimated ± 3.0 pp, high confidence |
| Qwen 3.6 Max (preview) | 42.9% | 47.9% | estimated ± 3.0 pp, high confidence |
| Qwen3.8 Max | 55.9% | 52.2% | estimated ± 3.0 pp, high confidence |
| Qwen3.8-Omni-Flash | 48.9% | 50.5% | estimated ± 3.0 pp, high confidence |
| Seed 2.1 Pro | 47.0% | 49.8% | estimated ± 3.0 pp, high confidence |
| Seed 2.1 Turbo | 43.7% | 48.3% | estimated ± 3.0 pp, high confidence |