Calibration
CharXiv → CharXiv w/o tools
CharXiv w/o tools is estimated from CharXiv with a offset logistic curve fitted on 12 models measured on both: y = 0.7715 + (1.2000 − 0.7715) / (1 + exp(−28.51·(x − 0.9704))), R² = 0.79, cross-validated error 2.3 pp. It is used for 7 estimates.
| Estimated model | CharXiv | CharXiv w/o tools | Source |
|---|---|---|---|
| Claude Sonnet 4.6 | 77.4% | 77.3% | estimated ± 2.3 pp, medium confidence |
| Gemini 3.1 Flash-Lite | 73.2% | 77.2% | estimated ± 2.3 pp, medium confidence |
| GLM-5.3-Flash | 89.4% | 81.5% | estimated ± 2.3 pp, high confidence |
| Muse Spark 1.1 | 88.4% | 80.5% | estimated ± 2.3 pp, high confidence |
| Nemotron 3 Nano Omni 30B A3B | 76.3% | 77.3% | estimated ± 2.3 pp, medium confidence |
| Sakana Fugu | 85.1% | 78.5% | estimated ± 2.3 pp, high confidence |
| Sakana Fugu-Ultra | 86.6% | 79.2% | estimated ± 2.3 pp, high confidence |