Calibration
HLE w/ tools → CyberGym
CyberGym is estimated from HLE w/ tools with a offset logistic curve fitted on 5 models measured on both: y = 0.7624 + (1.1656 − 0.7624) / (1 + exp(−20.43·(x − 0.6842))), R² = 0.97, cross-validated error 2.4 pp. It is used for 17 estimates.
| Estimated model | HLE w/ tools | CyberGym | Source |
|---|---|---|---|
| Agents-A1 | 47.6% | 76.8% | estimated ± 2.4 pp, medium confidence |
| Apodex 1.1 | 56.1% | 79.2% | estimated ± 2.4 pp, medium confidence |
| Claude Haiku 5.5 | 57.4% | 80.1% | estimated ± 2.4 pp, medium confidence |
| Claude Opus 5 | 64.7% | 89.1% | estimated ± 2.4 pp, low confidence |
| Claude Opus 5.5 | 67.7% | 94.9% | estimated ± 2.4 pp, low confidence |
| Claude Sonnet 5 | 57.4% | 80.1% | estimated ± 2.4 pp, medium confidence |
| Claude Sonnet 5.5 | 64.5% | 88.7% | estimated ± 2.4 pp, low confidence |
| dots3-note Preview | 52.6% | 77.8% | estimated ± 2.4 pp, medium confidence |
| GLM-5.3-Flash | 55.3% | 78.8% | estimated ± 2.4 pp, medium confidence |
| GPT-6 Astra | 57.2% | 79.9% | estimated ± 2.4 pp, medium confidence |
| Nemotron 3 Ultra | 37.4% | 76.3% | estimated ± 2.4 pp, low confidence |
| Ornith-1.5-35B-A3B | 33.4% | 76.3% | estimated ± 2.4 pp, low confidence |
| Ornith-1.5-397B | 56.1% | 79.2% | estimated ± 2.4 pp, medium confidence |
| Ornith-1.5-9B | 30.5% | 76.3% | estimated ± 2.4 pp, low confidence |
| Qwen3.7 Max | 53.5% | 78.1% | estimated ± 2.4 pp, medium confidence |
| Qwen3.8 Max | 56.2% | 79.3% | estimated ± 2.4 pp, medium confidence |
| Step 3.7 Flash | 47.2% | 76.8% | estimated ± 2.4 pp, medium confidence |