Calibration
Vals LiveCodeBench → NL2Repo
NL2Repo is estimated from Vals LiveCodeBench with a offset logistic curve fitted on 12 models measured on both: y = 0.4596 + (0.6498 − 0.4596) / (1 + exp(−200.00·(x − 0.8755))), R² = 0.31, cross-validated error 8.3 pp. It is used for 4 estimates.
| Estimated model | Vals LiveCodeBench | NL2Repo | Source |
|---|---|---|---|
| GPT-5.1-Codex | 85.6% | 46.3% | estimated ± 8.3 pp, low confidence |
| GPT-5.1-Codex-Max | 83.6% | 46.0% | estimated ± 8.3 pp, low confidence |
| GLM-4.6 | 81.0% | 46.0% | estimated ± 8.3 pp, low confidence |
| GLM-4.5 | 67.4% | 46.0% | estimated ± 8.3 pp, low confidence |
Fitted on
| Model | Vals LiveCodeBench | NL2Repo |
|---|---|---|
| DeepSeek V4 Flash 0731 | 87.3% | 54.2% |
| DeepSeek V4 Pro 0813 | 87.5% | 61.5% |
| GLM-5.1 | 81.4% | 42.7% |
| GLM-5.2 | 69.5% | 48.9% |
| GLM-5.3 | 80.5% | 58.0% |
| GLM-5.3-Flash | 80.5% | 56.3% |
| MiniMax M2.7 | 79.9% | 39.8% |
| MiniMax M3 | 82.2% | 42.1% |
| Qwen3.6 Plus | 86.0% | 37.9% |
| Qwen3.7 Max | 87.1% | 47.2% |
| Qwen3.8-27B | 84.0% | 42.3% |
| Qwen3.8 Max | 87.9% | 55.9% |