benchgap
Calibration

MMLU-Redux → SuperGPQA

SuperGPQA is estimated from MMLU-Redux with a Hill curve fitted on 9 models measured on both: y = 0.0000 + (1.2000 − 0.0000)·x^6.00 / (0.89960^6.00 + x^6.00), R² = 0.96, cross-validated error 4.1 pp. It is used for 6 estimates.

Estimated modelMMLU-ReduxSuperGPQASource
DeepSeek-R1-052893.4%66.7%estimated ± 4.1 pp, high confidence
Ternary Bonsai 2 27B89.1%58.3%estimated ± 4.1 pp, high confidence
Mellum2.1-12B-A2.5B-Thinking87.8%55.6%estimated ± 4.1 pp, high confidence
Mellum2-12B-A2.5B-Thinking86.2%52.4%estimated ± 4.1 pp, high confidence
Qwen3 32B85.7%51.3%estimated ± 4.1 pp, high confidence
Mellum2-12B-A2.5B-Instruct78.1%36.0%estimated ± 4.1 pp, high confidence

Fitted on

ModelMMLU-ReduxSuperGPQA
Claude Opus 4.596.6%70.6%
Kimi K2.594.5%69.2%
MiniCPM5-1B70.1%23.1%
MiniCPM5-2B84.7%40.8%
Qwen3.5 397B94.9%70.4%
Qwen3.6-27B93.5%66.0%
Qwen3.6 Plus94.5%71.6%
Qwen3.7 Max95.0%73.6%
Qwen3.7 Plus94.5%71.4%