Calibration
MMLU-ProX → SWE Multilingual
SWE Multilingual is estimated from MMLU-ProX with a inverse Michaelis–Menten curve fitted on 7 models measured on both: y = 0.82449·(x − 0.0000) / (0.0000 + 1.7836 − x), R² = 0.65, cross-validated error 2.6 pp. It is used for 8 estimates.
| Estimated model | MMLU-ProX | SWE Multilingual | Source |
|---|---|---|---|
| DeepSeek V3 0324 | 70.5% | 53.9% | estimated ± 2.6 pp, low confidence |
| GPT-4.1 | 72.7% | 56.7% | estimated ± 2.6 pp, low confidence |
| GPT-4o | 61.1% | 43.0% | estimated ± 2.6 pp, low confidence |
| Phi-4 | 49.9% | 32.0% | estimated ± 2.6 pp, low confidence |
| Qwen3 235B 2507 | 79.4% | 66.2% | estimated ± 2.6 pp, low confidence |
| Qwen3.5-122B-A10B | 82.2% | 70.5% | estimated ± 2.6 pp, low confidence |
| Qwen3.5-27B | 82.2% | 70.5% | estimated ± 2.6 pp, low confidence |
| Qwen3.5-35B-A3B | 81.0% | 68.6% | estimated ± 2.6 pp, low confidence |