Calibration
CyberGym → AutomationBench
AutomationBench is estimated from CyberGym with a offset logistic curve fitted on 10 models measured on both: y = 0.2844 + (0.5326 − 0.2844) / (1 + exp(−198.17·(x − 0.8413))), R² = 0.96, cross-validated error 5.8 pp. It is used for 18 estimates.
| Estimated model | CyberGym | AutomationBench | Source |
|---|---|---|---|
| Claude Mythos 5 | 83.8% | 36.9% | estimated ± 5.8 pp, medium confidence |
| Claude Mythos Preview | 83.1% | 31.3% | estimated ± 5.8 pp, medium confidence |
| Claude Opus 4.5 | 50.6% | 28.4% | estimated ± 5.8 pp, low confidence |
| Claude Opus 4.6 | 66.6% | 28.4% | estimated ± 5.8 pp, low confidence |
| Claude Opus 4.7 (Adaptive) | 73.1% | 28.4% | estimated ± 5.8 pp, low confidence |
| Claude Sonnet 4.6 | 65.2% | 28.4% | estimated ± 5.8 pp, low confidence |
| Gemini 3.5 Flash Cyber | 83.2% | 31.8% | estimated ± 5.8 pp, medium confidence |
| Gemini 3.8 Flash Cyber | 86.2% | 52.9% | estimated ± 5.8 pp, medium confidence |
| GLM-5 | 43.2% | 28.4% | estimated ± 5.8 pp, low confidence |
| GLM-5.1 | 68.7% | 28.4% | estimated ± 5.8 pp, low confidence |
| GPT-5.4 | 79.0% | 28.4% | estimated ± 5.8 pp, medium confidence |
| GPT-5.5 | 81.8% | 28.7% | estimated ± 5.8 pp, medium confidence |
| GPT-5.6 Luna | 77.9% | 28.4% | estimated ± 5.8 pp, medium confidence |
| GPT-5.6 Sol | 84.5% | 45.2% | estimated ± 5.8 pp, medium confidence |
| GPT-5.6 Terra | 81.8% | 28.7% | estimated ± 5.8 pp, medium confidence |
| Muse Spark | 43.5% | 28.4% | estimated ± 5.8 pp, low confidence |
| Muse Spark 1.1 | 59.0% | 28.4% | estimated ± 5.8 pp, low confidence |
| Fugu Cyber | 86.9% | 53.2% | estimated ± 5.8 pp, medium confidence |