Calibration
Chess Puzzles → FACTS Multimodal
FACTS Multimodal is estimated from Chess Puzzles with a linear curve fitted on 20 models measured on both: y = 0.3666·x + 0.2887, R² = 0.52, cross-validated error 5.6 pp. It is used for 26 estimates.
| Estimated model | Chess Puzzles | FACTS Multimodal | Source |
|---|---|---|---|
| GPT-5.5 Pro | 62.0% | 51.6% | estimated ± 5.6 pp, medium confidence |
| GPT-5.4 Pro | 59.0% | 50.5% | estimated ± 5.6 pp, medium confidence |
| GPT-5.4 nano | 45.0% | 45.4% | estimated ± 5.6 pp, medium confidence |
| GPT-5.1 | 42.0% | 44.3% | estimated ± 5.6 pp, medium confidence |
| Claude Opus 4.7 | 41.0% | 43.9% | estimated ± 5.6 pp, medium confidence |
| GPT-5 | 36.0% | 42.1% | estimated ± 5.6 pp, medium confidence |
| o3 | 36.0% | 42.1% | estimated ± 5.6 pp, medium confidence |
| DeepSeek V4 Flash 0731 | 32.0% | 40.6% | estimated ± 5.6 pp, medium confidence |
| Seed 2.1 Turbo | 26.0% | 38.4% | estimated ± 5.6 pp, medium confidence |
| GLM-4.7 | 21.0% | 36.6% | estimated ± 5.6 pp, medium confidence |
| GLM-5.3-Flash | 21.0% | 36.6% | estimated ± 5.6 pp, medium confidence |
| Inkling | 21.0% | 36.6% | estimated ± 5.6 pp, medium confidence |
| Seed 2.1 Pro | 21.0% | 36.6% | estimated ± 5.6 pp, medium confidence |
| o3-mini | 15.0% | 34.4% | estimated ± 5.6 pp, low confidence |
| Kimi K2.7 Code | 14.0% | 34.0% | estimated ± 5.6 pp, low confidence |
| Qwen3.5 Flash | 14.0% | 34.0% | estimated ± 5.6 pp, low confidence |
| o4-mini | 13.0% | 33.6% | estimated ± 5.6 pp, low confidence |
| GPT-4.1 | 11.0% | 32.9% | estimated ± 5.6 pp, low confidence |
| o1 | 10.0% | 32.5% | estimated ± 5.6 pp, low confidence |
| GPT-4o | 9.0% | 32.2% | estimated ± 5.6 pp, low confidence |
| Qwen3.5 Plus | 9.0% | 32.2% | estimated ± 5.6 pp, low confidence |
| Claude Haiku 4.5 | 8.0% | 31.8% | estimated ± 5.6 pp, low confidence |
| Claude 3 Opus | 4.0% | 30.3% | estimated ± 5.6 pp, low confidence |
| GPT-4o mini | 3.0% | 30.0% | estimated ± 5.6 pp, low confidence |
| GPT-4.1 mini | 1.0% | 29.2% | estimated ± 5.6 pp, low confidence |
| GPT-4.1 nano | 0.0% | 28.9% | estimated ± 5.6 pp, low confidence |
Fitted on
| Model | Chess Puzzles | FACTS Multimodal |
|---|---|---|
| Claude Haiku 5.5 | 55.0% | 38.9% |
| Claude Opus 5.5 | 67.0% | 56.4% |
| Claude Sonnet 5.5 | 59.0% | 50.4% |
| Gemini 3.1 Pro | 51.0% | 50.4% |
| Gemini 3.5 Flash-Lite | 16.0% | 36.1% |
| Gemini 3.6 Flash | 40.0% | 53.7% |
| Gemini 3.8 Flash | 58.0% | 57.6% |
| Gemini 3 Flash | 40.0% | 49.7% |
| Gemini 4 Argon | 59.0% | 54.2% |
| Gemini 4 Carbon | 78.0% | 60.2% |
| GPT-5.6 Luna | 43.0% | 37.4% |
| GPT-5.6 Sol | 59.0% | 46.4% |
| GPT-5.6 Terra | 47.0% | 39.9% |
| GPT-6.1 Sol | 68.0% | 52.6% |
| GPT-6 Astra | 74.0% | 55.2% |
| GPT-6 Luna | 36.0% | 38.7% |
| GPT-6 Sol | 56.0% | 43.9% |
| Grok 4.6 | 33.0% | 39.3% |
| Grok 4.7 | 41.0% | 40.9% |
| Qwen3.8 Max | 46.0% | 51.5% |