Calibration
AutomationBench → CWE-bench
CWE-bench is estimated from AutomationBench with a Hill curve fitted on 9 models measured on both: y = 0.2910 + (0.6325 − 0.2910)·x^6.00 / (0.28410^6.00 + x^6.00), R² = 0.32, cross-validated error 11.0 pp. It is used for 10 estimates.
| Estimated model | AutomationBench | CWE-bench | Source |
|---|---|---|---|
| ERNIE 5.5 Flash Preview | 61.9% | 62.9% | estimated ± 11.0 pp, low confidence |
| Atria Dawn Preview | 53.8% | 62.5% | estimated ± 11.0 pp, low confidence |
| Ling 3.1 Flash | 52.5% | 62.4% | estimated ± 11.0 pp, low confidence |
| Hy4 flash | 52.2% | 62.4% | estimated ± 11.0 pp, low confidence |
| LongCat-2.5 | 51.8% | 62.3% | estimated ± 11.0 pp, low confidence |
| Ultramarine 1 Flash | 50.2% | 62.2% | estimated ± 11.0 pp, low confidence |
| LFM3-1.7B | 41.3% | 60.0% | estimated ± 11.0 pp, low confidence |
| Ripple-65B Preview | 36.4% | 56.9% | estimated ± 11.0 pp, low confidence |
| DeepSeek V4 Pro | 31.8% | 51.7% | estimated ± 11.0 pp, low confidence |
| LFM2.5-24B-A2B | 27.7% | 44.9% | estimated ± 11.0 pp, low confidence |
Fitted on
| Model | AutomationBench | CWE-bench |
|---|---|---|
| Claude Fable 5.2 | 51.4% | 75.0% |
| Claude Opus 5.5 | 40.0% | 67.1% |
| DeepSeek V4.1 Flash 1017 | 60.5% | 57.6% |
| Gemini 4 Argon | 51.3% | 65.0% |
| GLM-5.3 | 48.2% | 42.0% |
| GPT-6 Astra | 41.4% | 64.2% |
| Hy4 | 56.2% | 64.1% |
| Hy4 preview | 32.1% | 47.3% |
| Qwen3.8 Max | 27.3% | 45.4% |