Calibration
AA-HLE → CritPt
CritPt is estimated from AA-HLE with a offset logistic curve fitted on 186 models measured on both: y = 0.0019 + (0.3319 − 0.0019) / (1 + exp(−15.66·(x − 0.4217))), R² = 0.91, cross-validated error 2.3 pp. It is used for 9 estimates.
| Estimated model | AA-HLE | CritPt | Source |
|---|---|---|---|
| Claude 3 Opus | 2.8% | 0.3% | estimated ± 2.3 pp, high confidence |
| DeepSeek R1 Distill Qwen 32B | 4.6% | 0.3% | estimated ± 2.3 pp, high confidence |
| Gemini 1.0 Pro | 4.2% | 0.3% | estimated ± 2.3 pp, high confidence |
| Gemini 1.5 Pro | 4.6% | 0.3% | estimated ± 2.3 pp, high confidence |
| GPT-4 Turbo | 3.1% | 0.3% | estimated ± 2.3 pp, high confidence |
| GPT-4o mini | 4.2% | 0.3% | estimated ± 2.3 pp, high confidence |
| o3-mini | 7.9% | 0.3% | estimated ± 2.3 pp, high confidence |
| Phi-4 Multimodal Instruct | 5.0% | 0.3% | estimated ± 2.3 pp, high confidence |
| Qwen2.5 Coder 32B Instruct | 3.5% | 0.3% | estimated ± 2.3 pp, high confidence |