Calibration
SWE-Atlas Codebase QnA → DeepSWE
DeepSWE is estimated from SWE-Atlas Codebase QnA with a Michaelis–Menten + offset curve fitted on 6 models measured on both: y = 0.0000 + 1.4005·x / (0.70693 + x), R² = 0.78, cross-validated error 8.5 pp. It is used for 0 estimates.
Fitted on
| Model | SWE-Atlas Codebase QnA | DeepSWE |
|---|---|---|
| Ling 3.1 Flash | 64.5% | 59.7% |
| Muse Spark 1.3 | 59.4% | 75.4% |
| Muse Spark Open | 58.0% | 60.0% |
| Beam 1.0 501B-A23B | 31.8% | 43.3% |
| Beam 1.0 501B-A23B FP8 | 32.2% | 43.0% |
| Beam 1.0 501B-A23B NVFP4 | 31.4% | 42.7% |