benchgap
Calibration

CharXiv → CharXiv w/o tools

CharXiv w/o tools is estimated from CharXiv with a offset logistic curve fitted on 14 models measured on both: y = 0.7759 + (1.2000 − 0.7759) / (1 + exp(−25.30·(x − 0.9772))), R² = 0.70, cross-validated error 2.6 pp. It is used for 4 estimates.

Estimated modelCharXivCharXiv w/o toolsSource
Muse Spark 1.188.4%81.3%estimated ± 2.6 pp, high confidence
Sakana Fugu-Ultra86.6%80.0%estimated ± 2.6 pp, high confidence
Sakana Fugu85.1%79.3%estimated ± 2.6 pp, high confidence
Nemotron 3 Nano Omni 30B A3B76.3%77.8%estimated ± 2.6 pp, medium confidence

Fitted on

ModelCharXivCharXiv w/o tools
Claude Mythos 593.5%88.9%
Claude Opus 4.791.0%82.1%
Claude Opus 4.889.9%80.5%
Claude Sonnet 588.3%77.0%
Gemini 3.7 Flash88.7%84.5%
Inkling82.0%78.1%
Inkling 2 Small87.7%85.6%
Inkling-Small81.3%77.4%
Kimi K3.1 Pro93.4%89.0%
Kimi K391.3%84.8%
Qwen3.8-27B90.2%83.7%
Qwen3.8-Flash-Next90.6%84.6%
Qwen3.8 Max93.5%88.4%
Qwen3.8-Omni-Flash91.4%83.5%