Calibration
τ²-bench results → HLE w/ tools
HLE w/ tools is estimated from τ²-bench results with a Hill curve fitted on 8 models measured on both: y = 0.0000 + (0.7350 − 0.0000)·x^6.00 / (0.83565^6.00 + x^6.00), R² = 0.67, cross-validated error 5.5 pp. It is used for 30 estimates.
| Estimated model | τ²-bench results | HLE w/ tools | Source |
|---|---|---|---|
| GLM-5-Turbo | 98.5% | 53.5% | estimated ± 5.5 pp, medium confidence |
| GLM-5V-Turbo | 98.5% | 53.5% | estimated ± 5.5 pp, medium confidence |
| Qwen 3.6 Max (preview) | 95.9% | 51.1% | estimated ± 5.5 pp, medium confidence |
| MiMo-V2-Pro | 95.0% | 50.2% | estimated ± 5.5 pp, medium confidence |
| Qwen3.5-27B | 93.9% | 49.1% | estimated ± 5.5 pp, medium confidence |
| Grok 4.1 Fast (Reasoning) | 93.3% | 48.5% | estimated ± 5.5 pp, medium confidence |
| GPT-5.2-Codex | 92.1% | 47.2% | estimated ± 5.5 pp, medium confidence |
| Qwen3.5-4B | 92.1% | 47.2% | estimated ± 5.5 pp, medium confidence |
| MiMo-V2-Omni | 91.2% | 46.2% | estimated ± 5.5 pp, medium confidence |
| Claude Opus 4.5 Thinking | 89.5% | 44.2% | estimated ± 5.5 pp, medium confidence |
| Qwen3.5-35B-A3B | 89.2% | 43.9% | estimated ± 5.5 pp, medium confidence |
| LFM2.5-8B-A1B | 88.1% | 42.5% | estimated ± 5.5 pp, medium confidence |
| Gemini 3 Pro | 87.1% | 41.3% | estimated ± 5.5 pp, medium confidence |
| GPT-5 (medium) | 86.5% | 40.5% | estimated ± 5.5 pp, medium confidence |
| GPT-5.3 Codex | 86.0% | 39.9% | estimated ± 5.5 pp, medium confidence |
| GPT-5.2 | 84.8% | 38.4% | estimated ± 5.5 pp, medium confidence |
| GPT-5.1-Codex | 83.0% | 36.0% | estimated ± 5.5 pp, low confidence |
| GPT-5.1-Codex-Max | 83.0% | 36.0% | estimated ± 5.5 pp, low confidence |
| o3 | 80.7% | 32.9% | estimated ± 5.5 pp, low confidence |
| LLaDA2.2-flash | 80.3% | 32.4% | estimated ± 5.5 pp, low confidence |
| Ternary Bonsai 2 27B | 80.2% | 32.3% | estimated ± 5.5 pp, low confidence |
| DeepSeek V3.2 | 78.9% | 30.5% | estimated ± 5.5 pp, low confidence |
| Agents-A1-4B | 78.2% | 29.5% | estimated ± 5.5 pp, low confidence |
| GLM-4.6 | 76.9% | 27.8% | estimated ± 5.5 pp, low confidence |
| Grok Code Fast 1 | 75.7% | 26.2% | estimated ± 5.5 pp, low confidence |
| Grok 4 | 74.9% | 25.1% | estimated ± 5.5 pp, low confidence |
| Qwen3 Max | 74.3% | 24.3% | estimated ± 5.5 pp, low confidence |
| Claude Opus 4.7 | 74.0% | 23.9% | estimated ± 5.5 pp, low confidence |
| Claude 4 Opus Thinking | 73.4% | 23.1% | estimated ± 5.5 pp, low confidence |
| Claude 4.1 Opus Thinking | 71.4% | 20.6% | estimated ± 5.5 pp, low confidence |
Fitted on
| Model | τ²-bench results | HLE w/ tools |
|---|---|---|
| DeepSeek V4 Pro 0813 | 96.2% | 60.0% |
| GLM-5 | 98.2% | 50.4% |
| GLM-5.1 | 97.7% | 52.3% |
| GLM-5.2 | 99.1% | 54.7% |
| Nemotron 3 Ultra | 83.3% | 37.4% |
| Qwen3.7 Max | 94.7% | 53.5% |
| Spark-X2.6-4B | 86.9% | 36.9% |
| Step 3.7 Flash | 98.5% | 47.2% |