Calibration
Gert Labs → OSWorld 2.1 Offline
OSWorld 2.1 Offline is estimated from Gert Labs with a Michaelis–Menten curve fitted on 10 models measured on both: y = 2.0000·x / (1.20027 + x), R² = 0.49, cross-validated error 10.1 pp. It is used for 25 estimates.
| Estimated model | Gert Labs | OSWorld 2.1 Offline | Source |
|---|---|---|---|
| Ember-1 | 58.0% | 65.2% | estimated ± 10.1 pp, low confidence |
| Muse Ember | 56.1% | 63.7% | estimated ± 10.1 pp, low confidence |
| GPT-5.3 Codex | 52.0% | 60.4% | estimated ± 10.1 pp, low confidence |
| Mercury 3 | 50.6% | 59.3% | estimated ± 10.1 pp, low confidence |
| Claude Opus 4.6 | 48.1% | 57.2% | estimated ± 10.1 pp, low confidence |
| GPT-5.2 | 47.7% | 56.8% | estimated ± 10.1 pp, low confidence |
| Qwen 3.6 Max (preview) | 47.5% | 56.7% | estimated ± 10.1 pp, low confidence |
| Gemini 3 Pro | 47.4% | 56.6% | estimated ± 10.1 pp, low confidence |
| Claude Sonnet 4.6 | 47.3% | 56.6% | estimated ± 10.1 pp, low confidence |
| Claude Opus 4.5 | 46.0% | 55.4% | estimated ± 10.1 pp, low confidence |
| Gemini 3.1 Pro | 45.7% | 55.1% | estimated ± 10.1 pp, low confidence |
| GPT-5.1 | 39.9% | 49.9% | estimated ± 10.1 pp, low confidence |
| Kimi K2.5 | 35.5% | 45.6% | estimated ± 10.1 pp, low confidence |
| Qwen3.7 Plus | 35.3% | 45.4% | estimated ± 10.1 pp, low confidence |
| Claude Sonnet 4.5 | 34.9% | 45.0% | estimated ± 10.1 pp, low confidence |
| Gemini 3 Flash | 34.2% | 44.4% | estimated ± 10.1 pp, low confidence |
| Muse Glimmer 30B | 32.7% | 42.8% | estimated ± 10.1 pp, low confidence |
| Gemini 2.5 Pro | 32.6% | 42.7% | estimated ± 10.1 pp, low confidence |
| Qwen3.6-27B | 32.4% | 42.5% | estimated ± 10.1 pp, low confidence |
| Grok 4 | 31.6% | 41.7% | estimated ± 10.1 pp, low confidence |
| Qwen3.6 Plus | 31.4% | 41.4% | estimated ± 10.1 pp, low confidence |
| Grok 4.20 Multi-agent | 30.2% | 40.2% | estimated ± 10.1 pp, low confidence |
| MiMo-V2.5 | 29.9% | 39.9% | estimated ± 10.1 pp, low confidence |
| Grok 4.1 Fast | 29.3% | 39.2% | estimated ± 10.1 pp, low confidence |
| Qwen3.5 397B | 29.2% | 39.2% | estimated ± 10.1 pp, low confidence |
Fitted on
| Model | Gert Labs | OSWorld 2.1 Offline |
|---|---|---|
| Claude Fable 5.1 | 77.0% | 83.5% |
| Claude Fable 5.2 | 88.2% | 89.3% |
| Claude Haiku 5.5 | 58.6% | 72.4% |
| Claude Opus 5.5 | 83.3% | 87.2% |
| Claude Sonnet 5.5 | 69.9% | 83.9% |
| Gemini 4 Argon | 80.6% | 74.7% |
| Gemini 4 Carbon | 83.7% | 81.5% |
| Gemini 4 Hydrogen | 79.1% | 77.6% |
| GPT-6.1 Terra | 83.2% | 74.2% |
| GPT-6 Luna | 58.8% | 48.9% |