benchgap
Calibration

ProgramBench → DeepSWE

DeepSWE is estimated from ProgramBench with a offset logistic curve fitted on 19 models measured on both: y = 0.6489 + (0.7293 − 0.6489) / (1 + exp(−200.00·(x − 0.2617))), R² = 0.46, cross-validated error 3.8 pp. It is used for 2 estimates.

Estimated modelProgramBenchDeepSWESource
GLM-5.263.7%72.9%estimated ± 3.8 pp, medium confidence
Kimi K2.7 Code53.6%72.9%estimated ± 3.8 pp, medium confidence

Fitted on

ModelProgramBenchDeepSWE
Claude Fable 5.187.6%67.4%
Claude Fable 5.285.6%74.3%
Claude Opus 593.0%68.8%
Claude Opus 5.591.2%74.2%
Claude Sonnet 5.579.7%71.0%
DeepSeek V4.1 Flash 101729.1%77.0%
ERNIE 5.5 Flash Preview29.5%71.2%
GLM-5.319.0%66.9%
Hy430.0%74.9%
Hy4 flash16.0%64.6%
Hy4 preview17.5%64.3%
Kimi K377.8%67.5%
LongCat-2.525.5%64.6%
MiMo-V2.6-Flash26.0%67.9%
MiMo-V2.6-Pro26.5%71.9%
Muse Wonder50.0%78.8%
Muse Wonder Full59.0%79.0%
Qwen4 Max31.5%75.7%
Step 5 Preview80.5%67.7%