benchgap
Calibration

MMLU → Vals GPQA Diamond

Vals GPQA Diamond is estimated from MMLU with a Hill curve fitted on 8 models measured on both: y = 0.0000 + (1.2000 − 0.0000)·x^6.00 / (0.85404^6.00 + x^6.00), R² = 0.73, cross-validated error 8.2 pp. It is used for 3 estimates.

Estimated modelMMLUVals GPQA DiamondSource
LongCat-Flash-Lite-Sparse85.3%59.8%estimated ± 8.2 pp, medium confidence
Mixtral 8x22B Instruct v0.177.8%43.6%estimated ± 8.2 pp, low confidence
Gemma 2 27B76.2%40.2%estimated ± 8.2 pp, low confidence

Fitted on

ModelMMLUVals GPQA Diamond
Claude 3.5 Haiku80.3%37.9%
GPT-4.190.2%65.4%
GPT-4.1 mini87.5%67.9%
GPT-4.1 nano80.1%50.8%
GPT-4o mini82.0%44.2%
o191.8%73.2%
o3-mini86.9%75.5%
o4-mini90.0%74.5%