benchgap
Calibration

MMMU-Pro → Video-MME (with subtitle)

Video-MME (with subtitle) is estimated from MMMU-Pro with a Michaelis–Menten curve fitted on 12 models measured on both: y = 1.4435·x / (0.51722 + x), R² = 0.88, cross-validated error 3.0 pp. It is used for 21 estimates.

Estimated modelMMMU-ProVideo-MME (with subtitle)Source
GPT-6.1 Terra85.5%89.9%estimated ± 3.0 pp, medium confidence
GPT-5.6 Sol83.0%88.9%estimated ± 3.0 pp, medium confidence
GPT-5.581.2%88.2%estimated ± 3.0 pp, high confidence
GPT-5.6 Terra80.7%88.0%estimated ± 3.0 pp, high confidence
Hy478.6%87.1%estimated ± 3.0 pp, high confidence
GPT-5.6 Luna78.4%87.0%estimated ± 3.0 pp, high confidence
Grok 4.378.1%86.8%estimated ± 3.0 pp, high confidence
Pareto 26.978.0%86.8%estimated ± 3.0 pp, high confidence
Claude Opus 4.677.3%86.5%estimated ± 3.0 pp, high confidence
Gemma 4 31B76.9%86.3%estimated ± 3.0 pp, high confidence
GPT-5.4 mini76.6%86.2%estimated ± 3.0 pp, high confidence
Step 5 Preview76.0%85.9%estimated ± 3.0 pp, high confidence
Hy4 flash75.8%85.8%estimated ± 3.0 pp, high confidence
Gemma 4 26B A4B73.8%84.9%estimated ± 3.0 pp, high confidence
Interfaze Beta71.1%83.6%estimated ± 3.0 pp, high confidence
GPT-5.4 nano66.1%81.0%estimated ± 3.0 pp, high confidence
Gemma 4 E4B52.6%72.8%estimated ± 3.0 pp, high confidence
Qwen2.5-VL-32B49.5%70.6%estimated ± 3.0 pp, high confidence
Mistral Small 3.149.3%70.4%estimated ± 3.0 pp, high confidence
Gemma 4 E2B44.2%66.5%estimated ± 3.0 pp, medium confidence
LFM2.5-VL-3B30.5%53.5%estimated ± 3.0 pp, medium confidence

Fitted on

ModelMMMU-ProVideo-MME (with subtitle)
Claude Opus 4.570.6%77.6%
Gemini 3 Pro81.0%88.4%
GPT-5.279.5%86.0%
Kimi K2.578.5%87.4%
LFM3-VL-2B45.7%69.0%
MiMo-V2.577.9%87.7%
MiniMax M378.1%85.4%
Qwen3.5 397B79.0%87.5%
Qwen3.6-27B75.8%87.7%
Qwen3.6-35B-A3B75.3%86.6%
Qwen3.7 Plus79.0%88.0%
Qwen3.8 Max82.3%90.4%