Calibration
AA AutomationBench → CWE-Bench-AA
CWE-Bench-AA is estimated from AA AutomationBench with a Hill curve fitted on 11 models measured on both: y = 0.0000 + (0.6745 − 0.0000)·x^6.00 / (0.44399^6.00 + x^6.00), R² = 0.63, cross-validated error 9.0 pp. It is used for 3 estimates.
| Estimated model | AA AutomationBench | CWE-Bench-AA | Source |
|---|---|---|---|
| Qwen4 Max | 72.6% | 64.1% | estimated ± 9.0 pp, medium confidence |
| Muse Ember | 68.1% | 62.6% | estimated ± 9.0 pp, medium confidence |
| Muse Wonder Full | 45.4% | 36.0% | estimated ± 9.0 pp, low confidence |
Fitted on
| Model | AA AutomationBench | CWE-Bench-AA |
|---|---|---|
| Claude Opus 5.5 | 69.5% | 58.3% |
| GLM-5.3 | 62.2% | 55.8% |
| GPT-6.2 Astra | 77.2% | 65.0% |
| GPT-6.2 Sol | 71.3% | 61.7% |
| GPT-6 Luna | 53.2% | 56.7% |
| Grok 4.7 | 65.6% | 68.3% |
| Kimi K3.1 Pro | 73.3% | 60.8% |
| MiMo-V2.6-Pro | 58.6% | 63.3% |
| Mistral Large 4 | 57.8% | 58.3% |
| Qwen3.8-27B | 48.2% | 26.7% |
| Beam 1.0 501B-A23B | 48.5% | 49.2% |