benchgap
Calibration

Chess Puzzles → FACTS Multimodal

FACTS Multimodal is estimated from Chess Puzzles with a linear curve fitted on 20 models measured on both: y = 0.3666·x + 0.2887, R² = 0.52, cross-validated error 5.6 pp. It is used for 26 estimates.

Estimated modelChess PuzzlesFACTS MultimodalSource
GPT-5.5 Pro62.0%51.6%estimated ± 5.6 pp, medium confidence
GPT-5.4 Pro59.0%50.5%estimated ± 5.6 pp, medium confidence
GPT-5.4 nano45.0%45.4%estimated ± 5.6 pp, medium confidence
GPT-5.142.0%44.3%estimated ± 5.6 pp, medium confidence
Claude Opus 4.741.0%43.9%estimated ± 5.6 pp, medium confidence
GPT-536.0%42.1%estimated ± 5.6 pp, medium confidence
o336.0%42.1%estimated ± 5.6 pp, medium confidence
DeepSeek V4 Flash 073132.0%40.6%estimated ± 5.6 pp, medium confidence
Seed 2.1 Turbo26.0%38.4%estimated ± 5.6 pp, medium confidence
GLM-4.721.0%36.6%estimated ± 5.6 pp, medium confidence
GLM-5.3-Flash21.0%36.6%estimated ± 5.6 pp, medium confidence
Inkling21.0%36.6%estimated ± 5.6 pp, medium confidence
Seed 2.1 Pro21.0%36.6%estimated ± 5.6 pp, medium confidence
o3-mini15.0%34.4%estimated ± 5.6 pp, low confidence
Kimi K2.7 Code14.0%34.0%estimated ± 5.6 pp, low confidence
Qwen3.5 Flash14.0%34.0%estimated ± 5.6 pp, low confidence
o4-mini13.0%33.6%estimated ± 5.6 pp, low confidence
GPT-4.111.0%32.9%estimated ± 5.6 pp, low confidence
o110.0%32.5%estimated ± 5.6 pp, low confidence
GPT-4o9.0%32.2%estimated ± 5.6 pp, low confidence
Qwen3.5 Plus9.0%32.2%estimated ± 5.6 pp, low confidence
Claude Haiku 4.58.0%31.8%estimated ± 5.6 pp, low confidence
Claude 3 Opus4.0%30.3%estimated ± 5.6 pp, low confidence
GPT-4o mini3.0%30.0%estimated ± 5.6 pp, low confidence
GPT-4.1 mini1.0%29.2%estimated ± 5.6 pp, low confidence
GPT-4.1 nano0.0%28.9%estimated ± 5.6 pp, low confidence

Fitted on

ModelChess PuzzlesFACTS Multimodal
Claude Haiku 5.555.0%38.9%
Claude Opus 5.567.0%56.4%
Claude Sonnet 5.559.0%50.4%
Gemini 3.1 Pro51.0%50.4%
Gemini 3.5 Flash-Lite16.0%36.1%
Gemini 3.6 Flash40.0%53.7%
Gemini 3.8 Flash58.0%57.6%
Gemini 3 Flash40.0%49.7%
Gemini 4 Argon59.0%54.2%
Gemini 4 Carbon78.0%60.2%
GPT-5.6 Luna43.0%37.4%
GPT-5.6 Sol59.0%46.4%
GPT-5.6 Terra47.0%39.9%
GPT-6.1 Sol68.0%52.6%
GPT-6 Astra74.0%55.2%
GPT-6 Luna36.0%38.7%
GPT-6 Sol56.0%43.9%
Grok 4.633.0%39.3%
Grok 4.741.0%40.9%
Qwen3.8 Max46.0%51.5%