Calibration
AA-HLE → CritPt
CritPt is estimated from AA-HLE with a offset logistic curve fitted on 200 models measured on both: y = 0.0002 + (0.3492 − 0.0002) / (1 + exp(−14.18·(x − 0.4275))), R² = 0.92, cross-validated error 3.4 pp. It is used for 9 estimates.
| Estimated model | AA-HLE | CritPt | Source |
|---|---|---|---|
| Inkling 2 Small | 44.1% | 19.1% | estimated ± 3.4 pp, high confidence |
| Phi-4 Multimodal Instruct | 5.0% | 0.2% | estimated ± 3.4 pp, high confidence |
| DeepSeek R1 Distill Qwen 32B | 4.6% | 0.2% | estimated ± 3.4 pp, high confidence |
| Gemini 1.5 Pro | 4.6% | 0.2% | estimated ± 3.4 pp, high confidence |
| Gemini 1.0 Pro | 4.2% | 0.2% | estimated ± 3.4 pp, high confidence |
| Qwen2.5 Coder 32B Instruct | 3.5% | 0.2% | estimated ± 3.4 pp, high confidence |
| Gemini 1.5 Flash | 3.2% | 0.1% | estimated ± 3.4 pp, high confidence |
| GPT-4 Turbo | 3.1% | 0.1% | estimated ± 3.4 pp, high confidence |
| Claude 3 Opus | 2.8% | 0.1% | estimated ± 3.4 pp, high confidence |