Calibration
HLE w/o tools → HLE
HLE is estimated from HLE w/o tools with a Michaelis–Menten curve fitted on 30 models measured on both: y = 2.0000·x / (1.21664 + x), R² = 0.83, cross-validated error 5.7 pp. It is used for 9 estimates.
| Estimated model | HLE w/o tools | HLE | Source |
|---|---|---|---|
| Claude Haiku 5.5 | 45.9% | 54.8% | estimated ± 5.7 pp, medium confidence |
| Claude Sonnet 5.5 | 56.9% | 63.7% | estimated ± 5.7 pp, medium confidence |
| Gemma 4 12B | 5.2% | 8.2% | estimated ± 5.7 pp, low confidence |
| Nemotron 3.5 Lightning 30B A3B NVFP4 | 10.5% | 15.8% | estimated ± 5.7 pp, medium confidence |
| Pareto 26.9 | 49.0% | 57.4% | estimated ± 5.7 pp, medium confidence |
| Beam | 36.2% | 45.9% | estimated ± 5.7 pp, medium confidence |
| Sakana Fugu | 47.2% | 55.9% | estimated ± 5.7 pp, medium confidence |
| Sakana Fugu-Ultra | 50.0% | 58.3% | estimated ± 5.7 pp, medium confidence |
| Solar Open 2 | 28.8% | 38.3% | estimated ± 5.7 pp, medium confidence |