benchgap
Calibration

AutomationBench → CWE-bench

CWE-bench is estimated from AutomationBench with a Hill curve fitted on 9 models measured on both: y = 0.2910 + (0.6325 − 0.2910)·x^6.00 / (0.28410^6.00 + x^6.00), R² = 0.32, cross-validated error 11.0 pp. It is used for 10 estimates.

Estimated modelAutomationBenchCWE-benchSource
ERNIE 5.5 Flash Preview61.9%62.9%estimated ± 11.0 pp, low confidence
Atria Dawn Preview53.8%62.5%estimated ± 11.0 pp, low confidence
Ling 3.1 Flash52.5%62.4%estimated ± 11.0 pp, low confidence
Hy4 flash52.2%62.4%estimated ± 11.0 pp, low confidence
LongCat-2.551.8%62.3%estimated ± 11.0 pp, low confidence
Ultramarine 1 Flash50.2%62.2%estimated ± 11.0 pp, low confidence
LFM3-1.7B41.3%60.0%estimated ± 11.0 pp, low confidence
Ripple-65B Preview36.4%56.9%estimated ± 11.0 pp, low confidence
DeepSeek V4 Pro31.8%51.7%estimated ± 11.0 pp, low confidence
LFM2.5-24B-A2B27.7%44.9%estimated ± 11.0 pp, low confidence

Fitted on

ModelAutomationBenchCWE-bench
Claude Fable 5.251.4%75.0%
Claude Opus 5.540.0%67.1%
DeepSeek V4.1 Flash 101760.5%57.6%
Gemini 4 Argon51.3%65.0%
GLM-5.348.2%42.0%
GPT-6 Astra41.4%64.2%
Hy456.2%64.1%
Hy4 preview32.1%47.3%
Qwen3.8 Max27.3%45.4%