benchgap
Calibration

Terminal-Bench 4.1 → Terminal-Bench Hard

Terminal-Bench Hard is estimated from Terminal-Bench 4.1 with a linear curve fitted on 5 models measured on both: y = 0.2775·x + 0.5245, R² = 0.90, cross-validated error 2.8 pp. It is used for 41 estimates.

Estimated modelTerminal-Bench 4.1Terminal-Bench HardSource
Claude Fable 5.265.8%70.7%estimated ± 2.8 pp, low confidence
Claude Opus 5.565.8%70.7%estimated ± 2.8 pp, low confidence
Gemini 4 Carbon64.6%70.4%estimated ± 2.8 pp, low confidence
Claude Sonnet 5.563.9%70.2%estimated ± 2.8 pp, low confidence
GPT-6.1 Sol60.9%69.4%estimated ± 2.8 pp, low confidence
GPT-6 Astra60.3%69.2%estimated ± 2.8 pp, low confidence
Kimi K3.1 Pro59.7%69.0%estimated ± 2.8 pp, low confidence
Claude Fable 5.157.6%68.4%estimated ± 2.8 pp, low confidence
Gemini 4 Hydrogen56.4%68.1%estimated ± 2.8 pp, low confidence
Gemini 4 Argon55.5%67.9%estimated ± 2.8 pp, low confidence
Claude Opus 553.3%67.2%estimated ± 2.8 pp, low confidence
GPT-6.1 Terra53.3%67.2%estimated ± 2.8 pp, low confidence
GPT-6 Sol50.3%66.4%estimated ± 2.8 pp, low confidence
Muse Wonder47.0%65.5%estimated ± 2.8 pp, low confidence
GLM-5.341.2%63.9%estimated ± 2.8 pp, medium confidence
Claude Haiku 5.540.0%63.6%estimated ± 2.8 pp, medium confidence
Grok 4.738.8%63.2%estimated ± 2.8 pp, medium confidence
DeepSeek V4.1 Flash 101738.5%63.1%estimated ± 2.8 pp, medium confidence
Hy437.9%63.0%estimated ± 2.8 pp, medium confidence
GLM-5.3-Flash37.0%62.7%estimated ± 2.8 pp, medium confidence
Muse Ember30.6%60.9%estimated ± 2.8 pp, medium confidence
Mistral Large 427.6%60.1%estimated ± 2.8 pp, medium confidence
MiMo-V2.6-Pro27.0%59.9%estimated ± 2.8 pp, medium confidence
Kimi K323.0%58.8%estimated ± 2.8 pp, medium confidence
Ember-121.8%58.5%estimated ± 2.8 pp, medium confidence
GPT-5.6 Terra21.5%58.4%estimated ± 2.8 pp, medium confidence
Grok 4.619.7%57.9%estimated ± 2.8 pp, medium confidence
Gemini 3.8 Flash18.2%57.5%estimated ± 2.8 pp, medium confidence
GPT-5.6 Luna17.6%57.3%estimated ± 2.8 pp, medium confidence
MiMo-V2.6-Flash16.1%56.9%estimated ± 2.8 pp, medium confidence
GPT-6 Luna15.2%56.7%estimated ± 2.8 pp, medium confidence
Muse Spark 1.314.9%56.6%estimated ± 2.8 pp, medium confidence
Gemini 3.7 Flash12.4%55.9%estimated ± 2.8 pp, medium confidence
Grok 4.512.4%55.9%estimated ± 2.8 pp, medium confidence
Claude Sonnet 512.1%55.8%estimated ± 2.8 pp, medium confidence
MiniMax M37.6%54.6%estimated ± 2.8 pp, medium confidence
DeepSeek V4 Pro6.7%54.3%estimated ± 2.8 pp, medium confidence
Beam 1.0 501B-A23B6.4%54.2%estimated ± 2.8 pp, medium confidence
Inkling-Small0.9%52.7%estimated ± 2.8 pp, low confidence
Nemotron 3.5 Super VL 120B A12B0.3%52.5%estimated ± 2.8 pp, low confidence
Inkling0.0%52.5%estimated ± 2.8 pp, low confidence

Fitted on

ModelTerminal-Bench 4.1Terminal-Bench Hard
Claude Fable 544.9%62.9%
Claude Opus 4.823.6%58.3%
Gemini 3.1 Pro2.7%53.8%
GPT-5.4 mini1.8%52.3%
GPT-5.6 Sol38.5%65.9%