benchgap
Calibration

AA-MMMU-Pro → OfficeQA Pro

OfficeQA Pro is estimated from AA-MMMU-Pro with a offset logistic curve fitted on 9 models measured on both: y = 0.5032 + (0.6762 − 0.5032) / (1 + exp(−200.00·(x − 0.8033))), R² = 0.71, cross-validated error 6.8 pp. It is used for 67 estimates.

Estimated modelAA-MMMU-ProOfficeQA ProSource
GPT-6.2 Astra89.5%67.6%estimated ± 6.8 pp, low confidence
GPT-6 Astra86.9%67.6%estimated ± 6.8 pp, medium confidence
GPT-6.1 Sol86.0%67.6%estimated ± 6.8 pp, medium confidence
Gemini 3.8 Flash85.6%67.6%estimated ± 6.8 pp, medium confidence
Gemini 3.7 Flash85.5%67.6%estimated ± 6.8 pp, medium confidence
GPT-6.1 Terra84.7%67.6%estimated ± 6.8 pp, medium confidence
Gemini 3.5 Flash84.3%67.6%estimated ± 6.8 pp, medium confidence
GPT-5.6 Sol83.4%67.6%estimated ± 6.8 pp, medium confidence
Gemini 3.6 Flash83.2%67.6%estimated ± 6.8 pp, medium confidence
GPT-6 Sol82.9%67.5%estimated ± 6.8 pp, medium confidence
Qwen3.8 Max Preview82.8%67.5%estimated ± 6.8 pp, medium confidence
Gemini 3.1 Pro82.4%67.3%estimated ± 6.8 pp, medium confidence
Muse Spark Open80.9%63.4%estimated ± 6.8 pp, medium confidence
GPT-5.6 Terra80.7%62.0%estimated ± 6.8 pp, medium confidence
Muse Spark80.5%60.4%estimated ± 6.8 pp, medium confidence
Qwen3.7 Plus80.5%60.4%estimated ± 6.8 pp, medium confidence
Grok 4.580.4%59.5%estimated ± 6.8 pp, medium confidence
Gemini 3 Pro80.2%57.8%estimated ± 6.8 pp, medium confidence
GPT-6 Luna79.7%54.1%estimated ± 6.8 pp, medium confidence
Kimi K2.679.4%52.6%estimated ± 6.8 pp, medium confidence
Apodex 1.179.2%51.9%estimated ± 6.8 pp, medium confidence
Apodex 1.1 Mini79.2%51.9%estimated ± 6.8 pp, medium confidence
MiniMax M3.1 Flash79.1%51.7%estimated ± 6.8 pp, medium confidence
Gemini 3.5 Flash-Lite79.0%51.4%estimated ± 6.8 pp, medium confidence
Ling 3.0 Flash VL79.0%51.4%estimated ± 6.8 pp, medium confidence
Gemini 3 Flash78.6%50.8%estimated ± 6.8 pp, medium confidence
GPT-5.6 Luna78.6%50.8%estimated ± 6.8 pp, medium confidence
Qwen3.8-Flash-Next78.6%50.8%estimated ± 6.8 pp, medium confidence
GPT-5.3 Codex78.5%50.7%estimated ± 6.8 pp, medium confidence
Grok 4.378.1%50.5%estimated ± 6.8 pp, medium confidence
Qwen3.6 Plus78.0%50.5%estimated ± 6.8 pp, medium confidence
Qwen3.5 397B77.3%50.4%estimated ± 6.8 pp, medium confidence
GPT-5.2-Codex76.3%50.3%estimated ± 6.8 pp, low confidence
Qwen3.8-27B76.3%50.3%estimated ± 6.8 pp, low confidence
GPT-5.175.5%50.3%estimated ± 6.8 pp, low confidence
Claude Opus 4.6 (Adaptive)75.4%50.3%estimated ± 6.8 pp, low confidence
Kimi K2.575.4%50.3%estimated ± 6.8 pp, low confidence
Step 3.7 Flash75.3%50.3%estimated ± 6.8 pp, low confidence
Qwen3.5-122B-A10B75.0%50.3%estimated ± 6.8 pp, low confidence
Qwen3.5-27B75.0%50.3%estimated ± 6.8 pp, low confidence
Qwen3.6-35B-A3B75.0%50.3%estimated ± 6.8 pp, low confidence
Gemini 2.5 Pro74.9%50.3%estimated ± 6.8 pp, low confidence
Qwen3.6-27B74.6%50.3%estimated ± 6.8 pp, low confidence
Muse Glimmer 30B74.3%50.3%estimated ± 6.8 pp, low confidence
GPT-574.2%50.3%estimated ± 6.8 pp, low confidence
Claude Opus 4.5 Thinking74.0%50.3%estimated ± 6.8 pp, low confidence
Inkling-Small74.0%50.3%estimated ± 6.8 pp, low confidence
Inkling73.5%50.3%estimated ± 6.8 pp, low confidence
Gemma 4 31B73.4%50.3%estimated ± 6.8 pp, low confidence
GPT-5.4 mini73.3%50.3%estimated ± 6.8 pp, low confidence
MiMo-V2.6-Flash73.1%50.3%estimated ± 6.8 pp, low confidence
GLM-5V-Turbo72.8%50.3%estimated ± 6.8 pp, low confidence
Qwen3.5-35B-A3B72.7%50.3%estimated ± 6.8 pp, low confidence
Claude Opus 4.672.5%50.3%estimated ± 6.8 pp, low confidence
GPT-5.1-Codex72.5%50.3%estimated ± 6.8 pp, low confidence
GPT-5.1-Codex-Max72.5%50.3%estimated ± 6.8 pp, low confidence
Claude Opus 4.571.2%50.3%estimated ± 6.8 pp, low confidence
Claude Sonnet 4.670.6%50.3%estimated ± 6.8 pp, low confidence
o370.1%50.3%estimated ± 6.8 pp, low confidence
MiMo-V2-Omni69.9%50.3%estimated ± 6.8 pp, low confidence
Gemma 4 12B69.7%50.3%estimated ± 6.8 pp, low confidence
Gemma 4 26B A4B69.2%50.3%estimated ± 6.8 pp, low confidence
o4-mini69.2%50.3%estimated ± 6.8 pp, low confidence
Grok 468.8%50.3%estimated ± 6.8 pp, low confidence
Gemini 2.5 Flash65.5%50.3%estimated ± 6.8 pp, low confidence
GPT-5.4 nano65.4%50.3%estimated ± 6.8 pp, low confidence
Qwen3.5-4B65.4%50.3%estimated ± 6.8 pp, low confidence

Fitted on

ModelAA-MMMU-ProOfficeQA Pro
Claude Opus 4.778.8%43.6%
Claude Opus 584.7%66.9%
Claude Opus 5.587.7%67.7%
GPT-5.478.4%53.2%
GPT-5.579.9%54.1%
Kimi K3.1 Pro83.9%67.4%
Kimi K380.5%63.3%
MiniMax M378.6%45.1%
Step 5 Preview76.4%60.3%