Calibration
τ³-bench results → PinchBench
PinchBench is estimated from τ³-bench results with a Michaelis–Menten + offset curve fitted on 7 models measured on both: y = 0.0000 + 0.9691·x / (0.01787 + x), R² = 0.83, cross-validated error 5.5 pp. It is used for 8 estimates.
| Estimated model | τ³-bench results | PinchBench | Source |
|---|---|---|---|
| Mercury 2.5 | 96.0% | 95.1% | estimated ± 5.5 pp, low confidence |
| Mistral Medium 3.5 128B | 91.4% | 95.0% | estimated ± 5.5 pp, low confidence |
| MiMo-V2.5-Pro | 72.9% | 94.6% | estimated ± 5.5 pp, low confidence |
| Qwen3.6 Plus | 70.7% | 94.5% | estimated ± 5.5 pp, low confidence |
| Claude Opus 4.5 | 70.2% | 94.5% | estimated ± 5.5 pp, low confidence |
| MiniMax M2.7 | 67.6% | 94.4% | estimated ± 5.5 pp, low confidence |
| Qwen3.6-35B-A3B | 67.2% | 94.4% | estimated ± 5.5 pp, low confidence |
| Mercury 3 | 30.2% | 91.5% | estimated ± 5.5 pp, low confidence |
Fitted on
| Model | τ³-bench results | PinchBench |
|---|---|---|
| LFM2.5-2.6B | 5.7% | 68.2% |
| LFM2.5-24B-A2B | 8.0% | 83.2% |
| Nemotron 3.5 Bolt 74B-A9B | 17.0% | 87.8% |
| Nemotron 3.5 Bolt 74B-A9B BF16 | 17.0% | 90.4% |
| Nemotron 3.5 Lightning 30B A3B NVFP4 | 9.5% | 83.4% |
| Nemotron 3 Ultra | 70.9% | 90.0% |
| Pokee-Isaac 28B | 66.2% | 95.7% |