Calibration
Vals LiveCodeBench → NL2Repo
NL2Repo is estimated from Vals LiveCodeBench with a offset logistic curve fitted on 12 models measured on both: y = 0.4596 + (0.6498 − 0.4596) / (1 + exp(−200.00·(x − 0.8755))), R² = 0.31, cross-validated error 8.3 pp. It is used for 4 estimates.
| Estimated model | Vals LiveCodeBench | NL2Repo | Source |
|---|---|---|---|
| GPT-5.1-Codex | 85.6% | 46.3% | estimated ± 8.3 pp, low confidence |
| GPT-5.1-Codex-Max | 83.6% | 46.0% | estimated ± 8.3 pp, low confidence |
| GLM-4.6 | 81.0% | 46.0% | estimated ± 8.3 pp, low confidence |
| GLM-4.5 | 67.4% | 46.0% | estimated ± 8.3 pp, low confidence |