benchgap
Calibration

CharXiv → VideoMMMU

VideoMMMU is estimated from CharXiv with a Hill curve fitted on 10 models measured on both: y = 0.8311 + (1.2000 − 0.8311)·x^6.00 / (1.27105^6.00 + x^6.00), R² = 0.45, cross-validated error 1.5 pp. It is used for 39 estimates.

Estimated modelCharXivVideoMMMUSource
Claude Mythos 593.5%88.2%estimated ± 1.5 pp, medium confidence
Kimi K3.1 Pro93.4%88.1%estimated ± 1.5 pp, medium confidence
Glimpse92.9%88.0%estimated ± 1.5 pp, medium confidence
Qwen3.8-Omni-Flash91.4%87.6%estimated ± 1.5 pp, medium confidence
Kimi K391.3%87.6%estimated ± 1.5 pp, medium confidence
Claude Opus 4.791.0%87.5%estimated ± 1.5 pp, medium confidence
Qwen3.8-Flash-Next90.6%87.4%estimated ± 1.5 pp, medium confidence
LongCat-2.590.2%87.3%estimated ± 1.5 pp, medium confidence
Qwen3.8-27B90.2%87.3%estimated ± 1.5 pp, medium confidence
Claude Opus 4.889.9%87.2%estimated ± 1.5 pp, medium confidence
GLM-5.3-Flash89.4%87.1%estimated ± 1.5 pp, medium confidence
Gemini 3.7 Flash88.7%86.9%estimated ± 1.5 pp, medium confidence
Muse Spark 1.188.4%86.9%estimated ± 1.5 pp, medium confidence
Claude Sonnet 588.3%86.8%estimated ± 1.5 pp, medium confidence
Inkling 2 Small87.7%86.7%estimated ± 1.5 pp, medium confidence
Sakana Fugu-Ultra86.6%86.5%estimated ± 1.5 pp, medium confidence
Muse Spark86.4%86.4%estimated ± 1.5 pp, medium confidence
Seed 2.1 Pro85.4%86.2%estimated ± 1.5 pp, medium confidence
Sakana Fugu85.1%86.2%estimated ± 1.5 pp, medium confidence
Gemini 3.5 Flash84.2%86.0%estimated ± 1.5 pp, medium confidence
GPT-5.482.8%85.7%estimated ± 1.5 pp, medium confidence
MiniMax M3.1 Flash82.5%85.7%estimated ± 1.5 pp, medium confidence
Seed 2.1 Turbo82.5%85.7%estimated ± 1.5 pp, medium confidence
Inkling82.0%85.6%estimated ± 1.5 pp, medium confidence
Inkling-Small81.3%85.5%estimated ± 1.5 pp, medium confidence
MiMo-V2.581.0%85.4%estimated ± 1.5 pp, medium confidence
Nemotron 3.5 Super VL 120B A12B80.8%85.4%estimated ± 1.5 pp, medium confidence
Kimi K2.680.4%85.3%estimated ± 1.5 pp, medium confidence
Gemini 3.1 Pro80.2%85.3%estimated ± 1.5 pp, medium confidence
Nemotron 3.5 Super VL 120B A12B NVFP479.9%85.3%estimated ± 1.5 pp, medium confidence
Qwen3.5-27B79.5%85.2%estimated ± 1.5 pp, medium confidence
Muse Glimmer 30B78.8%85.1%estimated ± 1.5 pp, medium confidence
Qwen3.5-35B-A3B77.5%84.9%estimated ± 1.5 pp, medium confidence
Claude Sonnet 4.677.4%84.9%estimated ± 1.5 pp, medium confidence
Qwen3.5-122B-A10B77.2%84.9%estimated ± 1.5 pp, medium confidence
Nemotron 3 Nano Omni 30B A3B76.3%84.8%estimated ± 1.5 pp, medium confidence
Gemini 3.1 Flash-Lite73.2%84.4%estimated ± 1.5 pp, medium confidence
Grok 4.2060.9%83.6%estimated ± 1.5 pp, low confidence
Command A+52.7%83.3%estimated ± 1.5 pp, low confidence

Fitted on

ModelCharXivVideoMMMU
Claude Opus 4.568.5%84.4%
Gemini 3 Pro81.4%87.6%
GPT-5.282.1%85.9%
Kimi K2.577.5%86.6%
Qwen3.5 397B80.8%84.7%
Qwen3.6-27B78.4%84.4%
Qwen3.6-35B-A3B78.0%83.7%
Qwen3.6 Plus81.5%84.0%
Qwen3.7 Plus85.9%85.4%
Qwen3.8 Max93.5%88.7%