benchgap
Calibration

Gert Labs → OSWorld 2.1 Offline

OSWorld 2.1 Offline is estimated from Gert Labs with a Michaelis–Menten curve fitted on 10 models measured on both: y = 2.0000·x / (1.20027 + x), R² = 0.49, cross-validated error 10.1 pp. It is used for 25 estimates.

Estimated modelGert LabsOSWorld 2.1 OfflineSource
Ember-158.0%65.2%estimated ± 10.1 pp, low confidence
Muse Ember56.1%63.7%estimated ± 10.1 pp, low confidence
GPT-5.3 Codex52.0%60.4%estimated ± 10.1 pp, low confidence
Mercury 350.6%59.3%estimated ± 10.1 pp, low confidence
Claude Opus 4.648.1%57.2%estimated ± 10.1 pp, low confidence
GPT-5.247.7%56.8%estimated ± 10.1 pp, low confidence
Qwen 3.6 Max (preview)47.5%56.7%estimated ± 10.1 pp, low confidence
Gemini 3 Pro47.4%56.6%estimated ± 10.1 pp, low confidence
Claude Sonnet 4.647.3%56.6%estimated ± 10.1 pp, low confidence
Claude Opus 4.546.0%55.4%estimated ± 10.1 pp, low confidence
Gemini 3.1 Pro45.7%55.1%estimated ± 10.1 pp, low confidence
GPT-5.139.9%49.9%estimated ± 10.1 pp, low confidence
Kimi K2.535.5%45.6%estimated ± 10.1 pp, low confidence
Qwen3.7 Plus35.3%45.4%estimated ± 10.1 pp, low confidence
Claude Sonnet 4.534.9%45.0%estimated ± 10.1 pp, low confidence
Gemini 3 Flash34.2%44.4%estimated ± 10.1 pp, low confidence
Muse Glimmer 30B32.7%42.8%estimated ± 10.1 pp, low confidence
Gemini 2.5 Pro32.6%42.7%estimated ± 10.1 pp, low confidence
Qwen3.6-27B32.4%42.5%estimated ± 10.1 pp, low confidence
Grok 431.6%41.7%estimated ± 10.1 pp, low confidence
Qwen3.6 Plus31.4%41.4%estimated ± 10.1 pp, low confidence
Grok 4.20 Multi-agent30.2%40.2%estimated ± 10.1 pp, low confidence
MiMo-V2.529.9%39.9%estimated ± 10.1 pp, low confidence
Grok 4.1 Fast29.3%39.2%estimated ± 10.1 pp, low confidence
Qwen3.5 397B29.2%39.2%estimated ± 10.1 pp, low confidence

Fitted on

ModelGert LabsOSWorld 2.1 Offline
Claude Fable 5.177.0%83.5%
Claude Fable 5.288.2%89.3%
Claude Haiku 5.558.6%72.4%
Claude Opus 5.583.3%87.2%
Claude Sonnet 5.569.9%83.9%
Gemini 4 Argon80.6%74.7%
Gemini 4 Carbon83.7%81.5%
Gemini 4 Hydrogen79.1%77.6%
GPT-6.1 Terra83.2%74.2%
GPT-6 Luna58.8%48.9%