Calibration
SWE Multilingual → INCLUDE
INCLUDE is estimated from SWE Multilingual with a linear curve fitted on 7 models measured on both: y = 0.2784·x + 0.6417, R² = 0.64, cross-validated error 2.0 pp. It is used for 41 estimates.
| Estimated model | SWE Multilingual | INCLUDE | Source |
|---|---|---|---|
| Claude Opus 5.5 | 93.9% | 90.3% | estimated ± 2.0 pp, low confidence |
| Claude Mythos 5 | 92.2% | 89.8% | estimated ± 2.0 pp, low confidence |
| Claude Sonnet 5.5 | 90.3% | 89.3% | estimated ± 2.0 pp, low confidence |
| Claude Fable 5.1 | 89.1% | 89.0% | estimated ± 2.0 pp, medium confidence |
| Ling 3.1 Flash | 88.5% | 88.8% | estimated ± 2.0 pp, medium confidence |
| Hy4 preview | 82.9% | 87.2% | estimated ± 2.0 pp, medium confidence |
| Qwen3.8-Flash-Next | 81.0% | 86.7% | estimated ± 2.0 pp, medium confidence |
| Qwen3.8-Omni-Flash | 80.5% | 86.6% | estimated ± 2.0 pp, medium confidence |
| Composer 2.5 | 79.8% | 86.4% | estimated ± 2.0 pp, medium confidence |
| Ornith-1.5-397B | 79.6% | 86.3% | estimated ± 2.0 pp, medium confidence |
| Ornith-1.0-397B | 78.9% | 86.1% | estimated ± 2.0 pp, medium confidence |
| Laguna S 2.1 | 78.5% | 86.0% | estimated ± 2.0 pp, medium confidence |
| Claude Sonnet 5 | 78.3% | 86.0% | estimated ± 2.0 pp, medium confidence |
| Beam 1.0 501B-A23B | 78.3% | 86.0% | estimated ± 2.0 pp, medium confidence |
| Beam 1.0 501B-A23B FP8 | 78.1% | 85.9% | estimated ± 2.0 pp, medium confidence |
| Grok 4.5 | 78.0% | 85.9% | estimated ± 2.0 pp, medium confidence |
| SWE-1.7 | 77.8% | 85.8% | estimated ± 2.0 pp, medium confidence |
| Beam 1.0 501B-A23B NVFP4 | 77.5% | 85.7% | estimated ± 2.0 pp, medium confidence |
| Kimi K2.6 | 76.7% | 85.5% | estimated ± 2.0 pp, medium confidence |
| MiniMax M2.7 | 76.5% | 85.5% | estimated ± 2.0 pp, medium confidence |
| DeepSeek V4 Pro 0813 | 76.2% | 85.4% | estimated ± 2.0 pp, medium confidence |
| dots3-note Preview | 75.7% | 85.2% | estimated ± 2.0 pp, medium confidence |
| Composer 2 | 73.7% | 84.7% | estimated ± 2.0 pp, medium confidence |
| DeepSeek V4 Flash 0731 | 73.3% | 84.6% | estimated ± 2.0 pp, medium confidence |
| Sage-A1.5 | 72.7% | 84.4% | estimated ± 2.0 pp, medium confidence |
| Ling 3.0 Flash | 72.4% | 84.3% | estimated ± 2.0 pp, medium confidence |
| MiMo-V2-Flash | 71.7% | 84.1% | estimated ± 2.0 pp, medium confidence |
| Ornith-1.5-35B-A3B | 71.4% | 84.0% | estimated ± 2.0 pp, medium confidence |
| Qwen3.6-27B | 71.3% | 84.0% | estimated ± 2.0 pp, medium confidence |
| Ornith-1.0-35B | 69.3% | 83.5% | estimated ± 2.0 pp, medium confidence |
| Nemotron 3.5 Super VL 120B A12B | 67.7% | 83.0% | estimated ± 2.0 pp, low confidence |
| Qwen3.6-35B-A3B | 67.2% | 82.9% | estimated ± 2.0 pp, low confidence |
| Nemotron 3.5 Super VL 120B A12B NVFP4 | 67.0% | 82.8% | estimated ± 2.0 pp, low confidence |
| Spark-X2.6-4B | 64.7% | 82.2% | estimated ± 2.0 pp, low confidence |
| Laguna M.1 | 63.1% | 81.7% | estimated ± 2.0 pp, low confidence |
| Laguna XS 2.1 | 63.1% | 81.7% | estimated ± 2.0 pp, low confidence |
| LongCat-Flash-Lite-Sparse | 59.3% | 80.7% | estimated ± 2.0 pp, low confidence |
| Laguna XS.2 | 57.7% | 80.2% | estimated ± 2.0 pp, low confidence |
| Qwen3-Coder 480B-A35B | 54.7% | 79.4% | estimated ± 2.0 pp, low confidence |
| Ornith-1.5-9B | 54.4% | 79.3% | estimated ± 2.0 pp, low confidence |
| Ornith-1.0-9B | 52.0% | 78.6% | estimated ± 2.0 pp, low confidence |
Fitted on
| Model | SWE Multilingual | INCLUDE |
|---|---|---|
| Claude Opus 4.5 | 77.5% | 86.2% |
| Claude Opus 4.8 | 84.4% | 87.6% |
| Claude Opus 5 | 89.5% | 89.8% |
| Kimi K2.5 | 73.0% | 83.3% |
| Qwen3.5 397B | 69.3% | 85.6% |
| Qwen3.7 Max | 78.3% | 86.2% |
| Qwen3.7 Plus | 75.8% | 83.0% |