Calibration
ProgramBench → DeepSWE
DeepSWE is estimated from ProgramBench with a offset logistic curve fitted on 19 models measured on both: y = 0.6489 + (0.7293 − 0.6489) / (1 + exp(−200.00·(x − 0.2617))), R² = 0.46, cross-validated error 3.8 pp. It is used for 2 estimates.
| Estimated model | ProgramBench | DeepSWE | Source |
|---|---|---|---|
| GLM-5.2 | 63.7% | 72.9% | estimated ± 3.8 pp, medium confidence |
| Kimi K2.7 Code | 53.6% | 72.9% | estimated ± 3.8 pp, medium confidence |
Fitted on
| Model | ProgramBench | DeepSWE |
|---|---|---|
| Claude Fable 5.1 | 87.6% | 67.4% |
| Claude Fable 5.2 | 85.6% | 74.3% |
| Claude Opus 5 | 93.0% | 68.8% |
| Claude Opus 5.5 | 91.2% | 74.2% |
| Claude Sonnet 5.5 | 79.7% | 71.0% |
| DeepSeek V4.1 Flash 1017 | 29.1% | 77.0% |
| ERNIE 5.5 Flash Preview | 29.5% | 71.2% |
| GLM-5.3 | 19.0% | 66.9% |
| Hy4 | 30.0% | 74.9% |
| Hy4 flash | 16.0% | 64.6% |
| Hy4 preview | 17.5% | 64.3% |
| Kimi K3 | 77.8% | 67.5% |
| LongCat-2.5 | 25.5% | 64.6% |
| MiMo-V2.6-Flash | 26.0% | 67.9% |
| MiMo-V2.6-Pro | 26.5% | 71.9% |
| Muse Wonder | 50.0% | 78.8% |
| Muse Wonder Full | 59.0% | 79.0% |
| Qwen4 Max | 31.5% | 75.7% |
| Step 5 Preview | 80.5% | 67.7% |