Score matrix
Every model × every benchmark
Every model on every benchmark: measured LLM scores and calibrated estimates for the missing ones, side by side.
Models
- 1-bit Bonsai 1.7B (1 measured, 2 estimated)
- 1-bit Bonsai 4B (1 measured, 2 estimated)
- 1-bit Bonsai 8B (1 measured, 2 estimated)
- A.X K2 (15 measured, 92 estimated)
- Agents-A1 (6 measured, 51 estimated)
- Agents-A1-4B (8 measured, 39 estimated)
- Alibaba GTE Reranker ModernBERT-base (1 measured, 0 estimated)
- Amazon Nova 2 Lite (1 measured, 6 estimated)
- Apodex 1.1 (18 measured, 119 estimated)
- Apodex 1.1 Mini (12 measured, 113 estimated)
- Atria Dawn Preview (11 measured, 58 estimated)
- AutoJev-27B (0 measured, 1 estimated)
- AutoJev-27B (RTX PRO 6000) (0 measured, 1 estimated)
- Autoloops Gemma 4 31B IT (1 measured, 0 estimated)
- BAAI bge-reranker-v2-m3 (1 measured, 0 estimated)
- Beam 1.0 501B-A23B (33 measured, 126 estimated)
- Beam 1.0 501B-A23B FP8 (19 measured, 109 estimated)
- Beam 1.0 501B-A23B NVFP4 (19 measured, 109 estimated)
- Bekko System One v0 17M (6 measured, 4 estimated)
- Bekko System One v0 400M (6 measured, 5 estimated)
- Bekko System One v0 68M (6 measured, 5 estimated)
- Bespoke Nimble 9B (1 measured, 0 estimated)
- Bespoke Nimble 9B v2 (6 measured, 5 estimated)
- Bespoke Nimble 9B v3 (6 measured, 5 estimated)
- Bev / Bonsai 27B (0 measured, 1 estimated)
- Blink v0.3 26B-A4B (NVFP4) (6 measured, 5 estimated)
- Bosun v3.1 0.6B (6 measured, 5 estimated)
- Bosun v3.1 1.7B (6 measured, 5 estimated)
- BTL-3 (2 measured, 11 estimated)
- BTL-4 (3 measured, 18 estimated)
- Celeris-1 (10 measured, 69 estimated)
- Celeris-1 Decision (4 measured, 7 estimated)
- Celeris-1.1 Decision (5 measured, 6 estimated)
- Certo v1 (1 measured, 0 estimated)
- classifier.dev (fast) (1 measured, 0 estimated)
- Claude 3 Haiku (9 measured, 44 estimated)
- Claude 3 Opus (5 measured, 56 estimated)
- Claude 3.5 Haiku (21 measured, 75 estimated)
- Claude 3.5 Sonnet (6 measured, 30 estimated)
- Claude 3.7 Sonnet (19 measured, 69 estimated)
- Claude 3.7 Sonnet Thinking (16 measured, 57 estimated)
- Claude 4 Opus (9 measured, 61 estimated)
- Claude 4 Opus Thinking (11 measured, 56 estimated)
- Claude 4 Sonnet (13 measured, 88 estimated)
- Claude 4.1 Opus (3 measured, 55 estimated)
- Claude Fable 5 (50 measured, 122 estimated)
- Claude Fable 5.1 (72 measured, 101 estimated)
- Claude Fable 5.2 (49 measured, 97 estimated)
- Claude Haiku 4.5 (18 measured, 96 estimated)
- Claude Haiku 4.5 Thinking (3 measured, 21 estimated)
- Claude Haiku 5.5 (48 measured, 114 estimated)
- Claude Mythos 5 (14 measured, 126 estimated)
- Claude Mythos 5.1 (2 measured, 18 estimated)
- Claude Mythos 5.2 (4 measured, 30 estimated)
- Claude Mythos Preview (3 measured, 39 estimated)
- Claude Opus 4.5 (80 measured, 102 estimated)
- Claude Opus 4.5 Thinking (20 measured, 95 estimated)
- Claude Opus 4.6 (53 measured, 125 estimated)
- Claude Opus 4.6 (Adaptive) (18 measured, 101 estimated)
- Claude Opus 4.7 (55 measured, 139 estimated)
- Claude Opus 4.8 (70 measured, 133 estimated)
- Claude Opus 5 (84 measured, 112 estimated)
- Claude Opus 5.5 (89 measured, 93 estimated)
- Claude Sonnet 4.5 (20 measured, 75 estimated)
- Claude Sonnet 4.5 Thinking (9 measured, 45 estimated)
- Claude Sonnet 4.6 (40 measured, 130 estimated)
- Claude Sonnet 5 (52 measured, 147 estimated)
- Claude Sonnet 5.5 (71 measured, 100 estimated)
- Clef (11 measured, 0 estimated)
- Clef-Flash (10 measured, 1 estimated)
- Clef-Omni (11 measured, 0 estimated)
- CLM-8B (1 measured, 0 estimated)
- CLM-v0.1-8B (6 measured, 0 estimated)
- Command A+ (17 measured, 107 estimated)
- Composer 2 (4 measured, 9 estimated)
- Composer 2 Fast (1 measured, 5 estimated)
- Composer 2.5 (7 measured, 32 estimated)
- Cygnet (1 measured, 0 estimated)
- d1 (5 measured, 6 estimated)
- d1-ultra (5 measured, 6 estimated)
- d1.2 (1 measured, 10 estimated)
- d1.2-ultra (1 measured, 10 estimated)
- Decider 4B (6 measured, 5 estimated)
- Decider chat · Gemma-4-31B (6 measured, 5 estimated)
- Decider chat · Qwen3.6-27B (6 measured, 5 estimated)
- decider-2b (7 measured, 5 estimated)
- decider-35b-a3b (7 measured, 5 estimated)
- decider-4b v2 (1 measured, 0 estimated)
- decisio 0.4.0 (Gemma 4 12B) (6 measured, 5 estimated)
- decisio 0.4.0 (Qwen3.6-35B-A3B) (6 measured, 5 estimated)
- Decision 2B v59 (1 measured, 0 estimated)
- Decision 4B v1.1 (0 measured, 1 estimated)
- Decision 4B v1.2 (0 measured, 1 estimated)
- Decision Fast v53a (1 measured, 0 estimated)
- decision-machine-1 (1 measured, 0 estimated)
- deck31b (Gemma 4 31B) (6 measured, 5 estimated)
- Deem 0.8B v1 (0 measured, 1 estimated)
- DeepSeek R1 Distill Llama 70B (4 measured, 20 estimated)
- DeepSeek R1 Distill Qwen 14B (4 measured, 21 estimated)
- DeepSeek R1 Distill Qwen 32B (5 measured, 29 estimated)
- DeepSeek V3 (21 measured, 76 estimated)
- DeepSeek V3 0324 (11 measured, 81 estimated)
- DeepSeek V3.1 (15 measured, 54 estimated)
- DeepSeek V3.2 (16 measured, 60 estimated)
- DeepSeek V4 Flash 0731 (46 measured, 120 estimated)
- DeepSeek V4 Pro (50 measured, 113 estimated)
- DeepSeek V4.1 Flash 1017 (56 measured, 129 estimated)
- DeepSeek-R1 (17 measured, 46 estimated)
- DeepSeek-R1-0528 (10 measured, 32 estimated)
- DeepSeek-R1-0528-Qwen3-8B (5 measured, 24 estimated)
- DeepSeek-V2.5 (2 measured, 23 estimated)
- Devstral Small (9 measured, 46 estimated)
- djev (7 measured, 5 estimated)
- DM-JEPA 1.1 (6 measured, 0 estimated)
- dots3-note Preview (28 measured, 107 estimated)
- Drex Max (10 measured, 1 estimated)
- Drex Max v1.1 (5 measured, 6 estimated)
- Drex v1.5 (10 measured, 1 estimated)
- Drex v2.0 (11 measured, 0 estimated)
- Eikos-27B (6 measured, 6 estimated)
- elephant-1 (11 measured, 0 estimated)
- Ember-1 (4 measured, 27 estimated)
- ERNIE 5.5 Flash Preview (14 measured, 80 estimated)
- Exaone 4.0 1.2B (6 measured, 32 estimated)
- Exaone 4.0 32B (7 measured, 35 estimated)
- EXAONE-4.0-1.2B-JEV v0.3 (6 measured, 5 estimated)
- ezjev 4B s2 (6 measured, 5 estimated)
- Flint (5 measured, 18 estimated)
- Fugu Cyber (1 measured, 18 estimated)
- Gemini 1.0 Pro (3 measured, 24 estimated)
- Gemini 1.5 Flash (12 measured, 50 estimated)
- Gemini 1.5 Pro (8 measured, 44 estimated)
- Gemini 2.0 Flash (17 measured, 64 estimated)
- Gemini 2.0 Flash-Lite (5 measured, 42 estimated)
- Gemini 2.5 Flash (11 measured, 59 estimated)
- Gemini 2.5 Flash-Lite (14 measured, 63 estimated)
- Gemini 2.5 Pro (27 measured, 116 estimated)
- Gemini 3 Flash (31 measured, 119 estimated)
- Gemini 3 Pro (46 measured, 93 estimated)
- Gemini 3.1 Flash-Lite (14 measured, 90 estimated)
- Gemini 3.1 Pro (60 measured, 135 estimated)
- Gemini 3.5 Flash (54 measured, 141 estimated)
- Gemini 3.5 Flash Cyber (1 measured, 18 estimated)
- Gemini 3.5 Flash-Lite (31 measured, 135 estimated)
- Gemini 3.6 Flash (33 measured, 142 estimated)
- Gemini 3.7 Flash (49 measured, 138 estimated)
- Gemini 3.8 Flash (64 measured, 122 estimated)
- Gemini 3.8 Flash Cyber (1 measured, 18 estimated)
- Gemini 4 Argon (55 measured, 107 estimated)
- Gemini 4 Carbon (37 measured, 91 estimated)
- Gemini 4 Hydrogen (24 measured, 90 estimated)
- Gemini 4 Neon (20 measured, 87 estimated)
- Gemini Decision 1 (1 measured, 10 estimated)
- Gemma 2 27B (1 measured, 8 estimated)
- Gemma 3 27B (13 measured, 79 estimated)
- Gemma 4 12B (21 measured, 100 estimated)
- Gemma 4 26B A4B (22 measured, 120 estimated)
- Gemma 4 31B (28 measured, 128 estimated)
- Gemma 4 E2B (14 measured, 84 estimated)
- Gemma 4 E4B (19 measured, 85 estimated)
- GEV-26B-Decide (6 measured, 5 estimated)
- Gevva E2B (6 measured, 5 estimated)
- Gevva E4B (6 measured, 5 estimated)
- GLiDE (4 measured, 7 estimated)
- GLiDE v1.2 (11 measured, 0 estimated)
- Glimpse (3 measured, 25 estimated)
- GLiNER2 large (1 measured, 0 estimated)
- GLiNER2.5 base (1 measured, 0 estimated)
- GLiNER2.5 multi (1 measured, 0 estimated)
- GLiNER2.5 small (1 measured, 0 estimated)
- GLiNER2.5-Decide (6 measured, 5 estimated)
- GLM-4.5 (4 measured, 24 estimated)
- GLM-4.5-Air (8 measured, 38 estimated)
- GLM-4.6 (12 measured, 57 estimated)
- GLM-4.6V (4 measured, 22 estimated)
- GLM-4.6V-Flash (4 measured, 20 estimated)
- GLM-4.7 (31 measured, 112 estimated)
- GLM-4.7-Flash (4 measured, 37 estimated)
- GLM-5 (45 measured, 98 estimated)
- GLM-5-Turbo (9 measured, 50 estimated)
- GLM-5.1 (42 measured, 122 estimated)
- GLM-5.2 (52 measured, 116 estimated)
- GLM-5.3 (61 measured, 97 estimated)
- GLM-5.3-Flash (43 measured, 113 estimated)
- GLM-5V-Turbo (11 measured, 73 estimated)
- GPT-4 Turbo (3 measured, 36 estimated)
- GPT-4.1 (21 measured, 90 estimated)
- GPT-4.1 mini (22 measured, 103 estimated)
- GPT-4.1 nano (18 measured, 92 estimated)
- GPT-4.5 (4 measured, 24 estimated)
- GPT-4o (15 measured, 85 estimated)
- GPT-4o mini (13 measured, 89 estimated)
- GPT-5 (24 measured, 121 estimated)
- GPT-5 mini (1 measured, 11 estimated)
- GPT-5.1 (25 measured, 124 estimated)
- GPT-5.1-Codex (13 measured, 95 estimated)
- GPT-5.1-Codex-Max (13 measured, 84 estimated)
- GPT-5.2 (57 measured, 122 estimated)
- GPT-5.2-Codex (17 measured, 101 estimated)
- GPT-5.3 Codex (19 measured, 105 estimated)
- GPT-5.3-Codex-Spark (1 measured, 20 estimated)
- GPT-5.4 (70 measured, 124 estimated)
- GPT-5.4 mini (41 measured, 140 estimated)
- GPT-5.4 nano (36 measured, 136 estimated)
- GPT-5.4 Pro (9 measured, 76 estimated)
- GPT-5.5 (76 measured, 121 estimated)
- GPT-5.5 Pro (9 measured, 76 estimated)
- GPT-5.6 Cyber (1 measured, 14 estimated)
- GPT-5.6 Luna (61 measured, 133 estimated)
- GPT-5.6 Sol (78 measured, 121 estimated)
- GPT-5.6 Terra (67 measured, 130 estimated)
- GPT-6 Astra (84 measured, 105 estimated)
- GPT-6 Cyber (4 measured, 26 estimated)
- GPT-6 Luna (54 measured, 121 estimated)
- GPT-6 Luna (Decisions) (1 measured, 10 estimated)
- GPT-6 Sol (52 measured, 118 estimated)
- GPT-6.1 Sol (55 measured, 113 estimated)
- GPT-6.1 Terra (41 measured, 117 estimated)
- GPT-6.2 Astra (41 measured, 101 estimated)
- GPT-6.2 Sol (32 measured, 83 estimated)
- GPT-OSS 120B (25 measured, 98 estimated)
- GPT-OSS 20B (19 measured, 80 estimated)
- Granite 4.2 30B (21 measured, 92 estimated)
- Granite 4.2 3B (17 measured, 74 estimated)
- Granite 4.2 8B (23 measured, 82 estimated)
- Granite-4.0-1B (5 measured, 15 estimated)
- Granite-4.0-350M (11 measured, 28 estimated)
- Granite-4.0-H-1B (11 measured, 30 estimated)
- Granite-4.0-H-350M (11 measured, 28 estimated)
- Grok 2 (4 measured, 34 estimated)
- Grok 2 mini (4 measured, 34 estimated)
- Grok 3 [Beta] (5 measured, 22 estimated)
- Grok 3 Mini (3 measured, 23 estimated)
- Grok 4 (20 measured, 89 estimated)
- Grok 4 Fast (Reasoning) (18 measured, 71 estimated)
- Grok 4.1 (1 measured, 12 estimated)
- Grok 4.1 Fast (13 measured, 62 estimated)
- Grok 4.1 Fast (Reasoning) (18 measured, 86 estimated)
- Grok 4.20 (26 measured, 129 estimated)
- Grok 4.20 Multi-agent (1 measured, 12 estimated)
- Grok 4.3 (31 measured, 142 estimated)
- Grok 4.5 (38 measured, 133 estimated)
- Grok 4.6 (46 measured, 110 estimated)
- Grok 4.7 (47 measured, 105 estimated)
- Grok Build 0.1 (1 measured, 14 estimated)
- Grok Code Fast 1 (11 measured, 55 estimated)
- Holo2-235B-A22B (1 measured, 12 estimated)
- Holo2-30B-A3B (1 measured, 11 estimated)
- Holo2-4B (1 measured, 11 estimated)
- Holo2-8B (1 measured, 11 estimated)
- Holo3-122B-A10B (1 measured, 15 estimated)
- Holo3-35B-A3B (1 measured, 15 estimated)
- Hopper (1 measured, 0 estimated)
- Hopper (G) 1.2 (6 measured, 5 estimated)
- Hopper 12B (trained) (6 measured, 5 estimated)
- Hy3 (11 measured, 103 estimated)
- Hy3 Preview (19 measured, 100 estimated)
- Hy4 (21 measured, 97 estimated)
- Hy4 flash (11 measured, 79 estimated)
- Hy4 preview (32 measured, 114 estimated)
- Imajev-4B (1 measured, 0 estimated)
- Imajev-4B (RTX 5090) (0 measured, 1 estimated)
- Inkling (36 measured, 156 estimated)
- Inkling 2 Mini (30 measured, 121 estimated)
- Inkling 2 Small (46 measured, 144 estimated)
- Inkling-Small (41 measured, 152 estimated)
- Instinct Dual 4B (0 measured, 1 estimated)
- Instinct Qwen3.8-27B (1 measured, 0 estimated)
- Intelif Qwen3-4B (6 measured, 5 estimated)
- Interfaze Beta (6 measured, 50 estimated)
- Intern-Decision-0.8B (6 measured, 5 estimated)
- Intern-Decision-2B (6 measured, 5 estimated)
- Intern-Decision-4B (6 measured, 5 estimated)
- JADE (6 measured, 5 estimated)
- Jais-3-3.7B (1 measured, 2 estimated)
- Jais-3-32B (2 measured, 19 estimated)
- Jais-3-375B-A23B (2 measured, 18 estimated)
- Jais-3-7B (2 measured, 16 estimated)
- Jais-3-MoVA-36B-A4B (2 measured, 19 estimated)
- jeff (7 measured, 4 estimated)
- Jev 1.13.0 (1 measured, 0 estimated)
- Jev 1.14 (11 measured, 0 estimated)
- JEV Qwen3.5-9B Base NVFP4 (1 measured, 0 estimated)
- JEV-27B (6 measured, 5 estimated)
- JEV-9B (6 measured, 5 estimated)
- jev-local Qwen3.5-9B (1 measured, 0 estimated)
- Jev-Omni (7 measured, 5 estimated)
- Jev-Style 0.8B v3 (6 measured, 4 estimated)
- JevAct (1 measured, 0 estimated)
- Jevfire (6 measured, 5 estimated)
- JevK5 9B v0.3.3 (6 measured, 5 estimated)
- JevK5 Lite (6 measured, 5 estimated)
- JevK5 v0.2.0 (1 measured, 0 estimated)
- JevK5 v0.3 (6 measured, 5 estimated)
- JevK5 v0.3 4B (0 measured, 1 estimated)
- JevOne (1 measured, 0 estimated)
- Jevstral 8B (6 measured, 5 estimated)
- Jobe (6 measured, 5 estimated)
- Jobe Qwen3.5-4B (1 measured, 0 estimated)
- JoshuaSP diffusiongemma (open-jev) (6 measured, 5 estimated)
- JPT-0.8B (6 measured, 5 estimated)
- JPT-35B-A3B (6 measured, 5 estimated)
- JPT-4B (6 measured, 5 estimated)
- JPT-9B (6 measured, 5 estimated)
- jqv Qwen3-32B (1 measured, 0 estimated)
- Julia 1 (6 measured, 4 estimated)
- K-Exaone (11 measured, 74 estimated)
- K-EXAONE 2.0 (22 measured, 94 estimated)
- Kanana-2 1.3B Instruct (3 measured, 1 estimated)
- Kanana-2 3B Instruct (3 measured, 1 estimated)
- Kas 4B (6 measured, 5 estimated)
- kev 0.5B (1 measured, 0 estimated)
- kev 0.6B (1 measured, 0 estimated)
- Kev 0.8B r15 (6 measured, 5 estimated)
- Kev 27B (6 measured, 5 estimated)
- kev 4B (1 measured, 0 estimated)
- Kev 4B r10 (6 measured, 5 estimated)
- kev 8B (1 measured, 0 estimated)
- Kev 9B v2 (6 measured, 5 estimated)
- Kimi K2 (21 measured, 48 estimated)
- Kimi K2.5 (77 measured, 102 estimated)
- Kimi K2.6 (57 measured, 142 estimated)
- Kimi K2.7 Code (21 measured, 124 estimated)
- Kimi K3 (76 measured, 120 estimated)
- Kimi K3.1 Pro (60 measured, 130 estimated)
- Kodiak v0.2 1B (accuracy) (6 measured, 5 estimated)
- Laguna M.1 (9 measured, 44 estimated)
- Laguna S 2.1 (5 measured, 38 estimated)
- Laguna XS 2.1 (4 measured, 30 estimated)
- Laguna XS.2 (9 measured, 42 estimated)
- Lavoir (6 measured, 4 estimated)
- Laya (7 measured, 5 estimated)
- Laya multilingual (0 measured, 1 estimated)
- Laya typed-decisions (0 measured, 1 estimated)
- lev (6 measured, 5 estimated)
- lev-350m (1 measured, 0 estimated)
- Lev1 3.7B (6 measured, 5 estimated)
- LFM2.5-2.6B (14 measured, 72 estimated)
- LFM2.5-2.6B-RLCD (6 measured, 5 estimated)
- LFM2.5-230M (6 measured, 16 estimated)
- LFM2.5-24B-A2B (12 measured, 42 estimated)
- LFM2.5-350M (3 measured, 17 estimated)
- LFM2.5-350M-RLCD (6 measured, 0 estimated)
- LFM2.5-8B-A1B (13 measured, 38 estimated)
- LFM2.5-VL-1.6B-Extract (7 measured, 39 estimated)
- LFM2.5-VL-3B (10 measured, 24 estimated)
- LFM2.5-VL-450M (7 measured, 34 estimated)
- LFM3-1.7B (19 measured, 59 estimated)
- LFM3-VL-2B (20 measured, 8 estimated)
- Ling 2.6 Flash (13 measured, 72 estimated)
- Ling 3.0 Flash (35 measured, 106 estimated)
- Ling 3.0 Flash Fin (4 measured, 16 estimated)
- Ling 3.0 Flash FP8 (4 measured, 32 estimated)
- Ling 3.0 Flash Sante (7 measured, 53 estimated)
- Ling 3.0 Flash VL (7 measured, 106 estimated)
- Ling 3.0 Tiny (7 measured, 68 estimated)
- Ling 3.1 Flash (23 measured, 113 estimated)
- LitJev Qwen3.8-27B (1 measured, 0 estimated)
- LLaDA2.2-flash (8 measured, 41 estimated)
- LLaDA2.2-mini (9 measured, 31 estimated)
- Llama 3.1 405B (7 measured, 31 estimated)
- Llama 3.1 70B (10 measured, 32 estimated)
- Llama 3.1 8B (6 measured, 21 estimated)
- Llama 3.3 70B (11 measured, 54 estimated)
- Llama 4 Maverick (13 measured, 105 estimated)
- Llama 4 Scout (11 measured, 76 estimated)
- local-jev Qwen3.5-4B (1 measured, 0 estimated)
- LongCat-2.5 (16 measured, 80 estimated)
- LongCat-Flash-Lite-Sparse (15 measured, 76 estimated)
- Magistral Medium (13 measured, 45 estimated)
- Magistral Small (13 measured, 43 estimated)
- MAI Code 2 (2 measured, 19 estimated)
- MAI Code 2 Flash (2 measured, 19 estimated)
- MAI-Code-1.1-Flash (2 measured, 19 estimated)
- MAI-Thinking-1 (12 measured, 71 estimated)
- Malkuth-2B (1 measured, 0 estimated)
- Malkuth-4B (1 measured, 0 estimated)
- mammoth-1 (7 measured, 4 estimated)
- Manchego v2.1 (0 measured, 1 estimated)
- Mellum2-12B-A2.5B-Instruct (6 measured, 25 estimated)
- Mellum2-12B-A2.5B-Thinking (6 measured, 29 estimated)
- Mellum2.1-12B-A2.5B-Thinking (7 measured, 32 estimated)
- Mercury 2.5 (12 measured, 85 estimated)
- Mercury 3 (6 measured, 49 estimated)
- Mercury Decide (5 measured, 6 estimated)
- metask-jev-4b (7 measured, 5 estimated)
- Microsoft-Decision-1 (5 measured, 6 estimated)
- MiMo-V2-Flash (18 measured, 84 estimated)
- MiMo-V2-Omni (11 measured, 83 estimated)
- MiMo-V2-Pro (13 measured, 76 estimated)
- MiMo-V2.5 (13 measured, 86 estimated)
- MiMo-V2.5-Pro (29 measured, 114 estimated)
- MiMo-V2.6-Flash (24 measured, 131 estimated)
- MiMo-V2.6-Pro (34 measured, 104 estimated)
- mini-jev (6 measured, 5 estimated)
- MiniCPM5-1B (14 measured, 21 estimated)
- MiniCPM5-2B (24 measured, 70 estimated)
- MiniMax M1 80k (6 measured, 14 estimated)
- MiniMax M2.5 (1 measured, 11 estimated)
- MiniMax M2.7 (37 measured, 109 estimated)
- MiniMax M3 (45 measured, 145 estimated)
- MiniMax M3.1 Flash (22 measured, 139 estimated)
- MiniMax-Text-01 (4 measured, 16 estimated)
- MiniMax-VL-01 (2 measured, 21 estimated)
- Mirror (1 measured, 0 estimated)
- Mistral Large 2 (7 measured, 35 estimated)
- Mistral Large 3 (18 measured, 96 estimated)
- Mistral Large 4 (25 measured, 106 estimated)
- Mistral Medium 3 (10 measured, 51 estimated)
- Mistral Medium 3.5 128B (21 measured, 117 estimated)
- Mistral Small 3.1 (17 measured, 90 estimated)
- Mistral Small 4 (13 measured, 91 estimated)
- Mixedbread mxbai-rerank-base-v2 (1 measured, 0 estimated)
- Mixtral 8x22B Instruct v0.1 (1 measured, 8 estimated)
- mmastrac diffusiongemma (vLLM PR 57250) (6 measured, 5 estimated)
- Muse Ember (10 measured, 57 estimated)
- Muse Glimmer 30B (25 measured, 133 estimated)
- Muse Spark (36 measured, 112 estimated)
- Muse Spark 1.1 (34 measured, 135 estimated)
- Muse Spark 1.2 (23 measured, 119 estimated)
- Muse Spark 1.3 (35 measured, 120 estimated)
- Muse Spark Open (22 measured, 142 estimated)
- Muse Wonder (33 measured, 110 estimated)
- Muse Wonder Full (21 measured, 95 estimated)
- Needle 3 (2-bit) (0 measured, 1 estimated)
- Needle 3 (options as tools) (0 measured, 1 estimated)
- Nemotron 3 Nano 30B (14 measured, 72 estimated)
- Nemotron 3 Nano Omni 30B A3B (23 measured, 92 estimated)
- Nemotron 3 Super 100B (7 measured, 37 estimated)
- Nemotron 3 Ultra (40 measured, 121 estimated)
- Nemotron 3.5 Bolt 74B-A9B (12 measured, 93 estimated)
- Nemotron 3.5 Bolt 74B-A9B BF16 (9 measured, 67 estimated)
- Nemotron 3.5 Bolt E4B (3 measured, 8 estimated)
- Nemotron 3.5 Bolt E4B BF16 (2 measured, 6 estimated)
- Nemotron 3.5 Lightning 30B A3B NVFP4 (22 measured, 77 estimated)
- Nemotron 3.5 Super VL 120B A12B (24 measured, 129 estimated)
- Nemotron 3.5 Super VL 120B A12B NVFP4 (23 measured, 128 estimated)
- Nemotron Diffusion 8B (optimized vLLM) (0 measured, 1 estimated)
- Nemotron Ultra 253B (10 measured, 33 estimated)
- NeoHorse-Jev-4B (6 measured, 5 estimated)
- NInfer Qwen3.8-27B NVFP4 (1 measured, 0 estimated)
- NInfer Qwen3.8-27B NVFP4 (T=1.5) (1 measured, 0 estimated)
- NInfer Qwen3.8-Flash-Next mixed (1 measured, 0 estimated)
- North Micro Vision Instruct (1 measured, 7 estimated)
- North Mini Code (10 measured, 79 estimated)
- Nova Pro (9 measured, 43 estimated)
- o1 (17 measured, 76 estimated)
- o1-mini (7 measured, 26 estimated)
- o1-preview (3 measured, 39 estimated)
- o1-pro (2 measured, 25 estimated)
- o3 (16 measured, 97 estimated)
- o3-mini (14 measured, 63 estimated)
- o3-pro (2 measured, 25 estimated)
- o4-mini (23 measured, 96 estimated)
- Open Jev JSON Canvas (1 measured, 0 estimated)
- open-alternative-jev Qwen3.5-4B (1 measured, 0 estimated)
- open-jev (pngwn) (6 measured, 5 estimated)
- Open-Jev 2B (1 measured, 0 estimated)
- Open-Jev 9B (1 measured, 0 estimated)
- open-jev-deberta-v3-large (1 measured, 0 estimated)
- OpenDecision ModernBERT-large (1 measured, 0 estimated)
- OpenJev (1 measured, 0 estimated)
- OpenJev DiffusionGemma 26B-A4B NVFP4 (1 measured, 0 estimated)
- openJev Verdict (1 measured, 0 estimated)
- openJev Verdict 1.4 (1 measured, 0 estimated)
- openjev-sglang Qwen3.6-35B-A3B (1 measured, 0 estimated)
- OpenSourceJev Qwen3.5-4B Q4_K_M (1 measured, 0 estimated)
- openvons (6 measured, 5 estimated)
- OrcaCyber-Nano-1.0 (6 measured, 30 estimated)
- Ornith-1.0-35B (6 measured, 38 estimated)
- Ornith-1.0-397B (6 measured, 38 estimated)
- Ornith-1.0-9B (6 measured, 35 estimated)
- Ornith-1.5-35B-A3B (17 measured, 89 estimated)
- Ornith-1.5-397B (17 measured, 97 estimated)
- Ornith-1.5-9B (15 measured, 80 estimated)
- Pareto 26.10 Preview (3 measured, 38 estimated)
- Pareto 26.9 (4 measured, 62 estimated)
- Particle-1 (4 measured, 40 estimated)
- Perplexity Decider Flash v1.2 4B (11 measured, 0 estimated)
- Perplexity Decider Flash v1.3 4B (7 measured, 4 estimated)
- Perplexity Decider v1.2 27B (11 measured, 0 estimated)
- Perplexity Decider v1.3 27B (7 measured, 4 estimated)
- Phi-3.5-mini (5 measured, 12 estimated)
- Phi-4 (9 measured, 30 estimated)
- Phi-4 Multimodal Instruct (3 measured, 21 estimated)
- Phi-4-reasoning (12 measured, 35 estimated)
- Phi-4-reasoning-plus (6 measured, 27 estimated)
- Plumb-4B (1 measured, 0 estimated)
- Pokee-Isaac 28B (5 measured, 13 estimated)
- Quasar 438B (9 measured, 94 estimated)
- Quyet-1.0-Large (6 measured, 5 estimated)
- Qwen 3.6 Max (preview) (18 measured, 90 estimated)
- Qwen-2.5-1B-RLCD (6 measured, 1 estimated)
- Qwen2.5 Coder 32B Instruct (3 measured, 24 estimated)
- Qwen2.5-Max (3 measured, 19 estimated)
- Qwen2.5-VL-32B (3 measured, 38 estimated)
- Qwen3 235B 2507 (9 measured, 37 estimated)
- Qwen3 235B-A22B (9 measured, 19 estimated)
- Qwen3 235B-A22B (Reasoning) (9 measured, 23 estimated)
- Qwen3 32B (9 measured, 20 estimated)
- Qwen3 32B (Reasoning) (9 measured, 20 estimated)
- Qwen3 Max (15 measured, 84 estimated)
- Qwen3-Coder 480B-A35B (2 measured, 14 estimated)
- Qwen3-Omni-30B-A3B-Instruct (7 measured, 40 estimated)
- Qwen3-Omni-30B-A3B-Thinking (7 measured, 45 estimated)
- Qwen3-Reranker-4B (1 measured, 0 estimated)
- Qwen3-VL-235B-A22B Instruct (8 measured, 23 estimated)
- Qwen3.5 397B (69 measured, 118 estimated)
- Qwen3.5 Flash (7 measured, 62 estimated)
- Qwen3.5 Plus (4 measured, 46 estimated)
- Qwen3.5-0.8B Decision Model (1 measured, 0 estimated)
- Qwen3.5-122B-A10B (30 measured, 140 estimated)
- Qwen3.5-27B (30 measured, 114 estimated)
- Qwen3.5-35B-A3B (28 measured, 103 estimated)
- Qwen3.5-4B (8 measured, 53 estimated)
- Qwen3.5-9B Jev-like data-mix v2 (1 measured, 0 estimated)
- Qwen3.6 Plus (62 measured, 133 estimated)
- Qwen3.6-27B (52 measured, 131 estimated)
- Qwen3.6-35B-A3B (54 measured, 130 estimated)
- Qwen3.7 Flash (3 measured, 25 estimated)
- Qwen3.7 Max (60 measured, 109 estimated)
- Qwen3.7 Plus (62 measured, 135 estimated)
- Qwen3.8 Max (67 measured, 121 estimated)
- Qwen3.8 Max Preview (13 measured, 110 estimated)
- Qwen3.8-27B (55 measured, 139 estimated)
- Qwen3.8-27B (Chutes TEE) (1 measured, 0 estimated)
- Qwen3.8-Flash-Next (33 measured, 123 estimated)
- Qwen3.8-Omni-Flash (16 measured, 74 estimated)
- Qwen4 Max (16 measured, 79 estimated)
- QwQ-32B (14 measured, 29 estimated)
- Raw Phi-4 mini direct logits (1 measured, 0 estimated)
- Raw Qwen3 0.6B direct logits (1 measured, 0 estimated)
- Raw Qwen3 1.7B direct logits (1 measured, 0 estimated)
- Raw Qwen3 4B Instruct 2507 direct logits (1 measured, 0 estimated)
- Raw Qwen3 8B direct logits (1 measured, 0 estimated)
- razorback16 openjev diffusiongemma (NVFP4, vLLM) (6 measured, 5 estimated)
- reflex 4B (1 measured, 0 estimated)
- reflex-27b (1 measured, 0 estimated)
- Ripple-65B Preview (17 measured, 93 estimated)
- Ryotide Qwen v0.3.1 (6 measured, 5 estimated)
- Sage-A1.5 (10 measured, 84 estimated)
- Sakana Fugu (10 measured, 71 estimated)
- Sakana Fugu-Ultra (10 measured, 71 estimated)
- Sarvam 105B (16 measured, 64 estimated)
- Sarvam 30B (14 measured, 61 estimated)
- Seed 2.1 Pro (10 measured, 67 estimated)
- Seed 2.1 Turbo (10 measured, 67 estimated)
- SemIf (6 measured, 5 estimated)
- SemIf Qwen3.5-4B (1 measured, 0 estimated)
- Sieve 2B (6 measured, 5 estimated)
- Sieve 9B (6 measured, 5 estimated)
- Sifr 0.8B v3.1 (6 measured, 5 estimated)
- simple-jev · Qwen3.8-27B (6 measured, 5 estimated)
- SimpleJev Qwen3.5-0.8B (1 measured, 0 estimated)
- SimpleJev Qwen3.6-35B-A3B (1 measured, 0 estimated)
- SimpleJev Qwen3.8-27B (1 measured, 0 estimated)
- smalljev semantic-v9 (1 measured, 0 estimated)
- Solar Open 2 (12 measured, 77 estimated)
- Solar Pro 2 (8 measured, 36 estimated)
- Solar Pro 3 (11 measured, 70 estimated)
- Solar Pro 4 (17 measured, 111 estimated)
- Solomon v1.1 (6 measured, 5 estimated)
- Soofi S 30B-A3B (4 measured, 15 estimated)
- spark-s1-4b-v6 (1 measured, 0 estimated)
- Spark-X2.6-4B (21 measured, 90 estimated)
- Standard One 8B (1 measured, 0 estimated)
- Step 3.5 Flash (3 measured, 19 estimated)
- Step 3.7 Flash (24 measured, 139 estimated)
- Step 5 Preview (39 measured, 139 estimated)
- SubQ Snap 1.1 (1 measured, 10 estimated)
- Surogate Rune 26B-A4B v3 (6 measured, 5 estimated)
- Surogate Rune 26B-A4B v3 (RTX PRO 6000) (0 measured, 1 estimated)
- swanOne (1 measured, 0 estimated)
- SWE-1.7 (4 measured, 24 estimated)
- SWE-2 (5 measured, 25 estimated)
- SWE-2 Flash (7 measured, 35 estimated)
- system-one Qwen3-8B (1 measured, 0 estimated)
- system-one-gemma (6 measured, 4 estimated)
- system-one-open (1 measured, 0 estimated)
- Ternary Bonsai 1.7B (1 measured, 2 estimated)
- Ternary Bonsai 2 27B (15 measured, 72 estimated)
- Ternary Bonsai 4B (1 measured, 2 estimated)
- Ternary Bonsai 8B (1 measured, 2 estimated)
- Tev1-0.8B-experimental (6 measured, 5 estimated)
- Tev1-4B-experimental (6 measured, 5 estimated)
- this-that 1.2 (6 measured, 5 estimated)
- Torchcast Decision 27B (6 measured, 5 estimated)
- Trinity-Large-Preview (13 measured, 82 estimated)
- Trinity-Large-Thinking (15 measured, 88 estimated)
- typecastlm (1 measured, 0 estimated)
- UI-Mate-27B (1 measured, 15 estimated)
- UI-Mate-9B (1 measured, 15 estimated)
- Ultramarine 1 Flash (4 measured, 39 estimated)
- Ultravox v0.6 Llama 3.3 70B (8 measured, 68 estimated)
- Verdict (6 measured, 0 estimated)
- verdict-small (1 measured, 0 estimated)
- vLLM-SR Decision 2.0 Eos 0.8B (6 measured, 5 estimated)
- vLLM-SR Decision 2.0 Kai 0.6B (6 measured, 5 estimated)
- vLLM-SR Decision 2.0 Lux 9B (6 measured, 5 estimated)
- vLLM-SR Decision 2.0 Nox 4B (6 measured, 5 estimated)
- vLLM-SR Decision 2.0 Sol 2B (6 measured, 5 estimated)
- vLLM-SR Decision 2.0 Vega 27B (6 measured, 5 estimated)
- Von 1.3 (6 measured, 5 estimated)
- Von 395M (1 measured, 0 estimated)
- Winnow-12B Q8 (7 measured, 5 estimated)
- Winnow-E4B (6 measured, 5 estimated)
- Xor 1.2 (6 measured, 5 estimated)
- ZAYA1-74B-Preview (6 measured, 32 estimated)
- ZAYA1-8B (10 measured, 34 estimated)
- ZAYA1-VL-8B (5 measured, 18 estimated)
- ZeroEntropy zerank-2 (1 measured, 0 estimated)
- Zev (6 measured, 0 estimated)
Benchmarks
Agentic · terminal
- AA Terminal-Bench 2.1
- AA Terminal-Bench 4.0
- Terminal-Bench 4.0
- Terminal-Bench 2.0
- Terminal-Bench 2.1
- Terminal-Bench 3.0
- Terminal-Bench Hard
- Terminal-Bench-Science 0.1
- Terminal-Bench 4.1
- Terminal-Bench 2.1 (Vals AI)
Agentic · tools
- AA Agentic Index
- AA-AnalystAgent
- AA AutomationBench
- CWE-Bench-AA
- AA Cyber Index
- DeepsecBench-AA
- AA EnterpriseOps-Gym
- GDP.pdf
- AA Harvey LAB v1.0
- Harvey LAB-AA v1.1
- AA ITBench
- AA Tau3 Banking
- Agents' Last Exam
- APEX-Agents
- APEX-Agents-AA
- ApprenticeBench
- AutomationBench
- BFCL v4
- BrowseComp
- Claw-Eval
- CWE-bench v1
- CWE-bench
- CyberGym
- DeepPlanning
- DeepSearchQA
- DRACO
- ExploitGym
- Finance Agent v2
- GDPval-AA
- Gert Labs
- Hermes Bench
- Hermes Index
- HLE w/ tools
- JobBench
- LifeVal
- MazeBench
- MCP Atlas
- MCP-Tasks
- MKA
- OSWorld 2.1
- OSWorld 2.1 Offline
- OSWorld-Verified
- OSWorld 2.0
- PinchBench
- QwenClawBench
- ResearchClawBench
- SkillsBench
- τ²-bench results
- τ³-bench results
- Toolathlon
- Toolathlon-Verified
- VITA-Bench
- WideResearch
Coding
- AA Coding Index
- AA LiveCodeBench
- AA-SciCode
- Bug Hunt Bench
- CursorBench 3.1
- CursorBench 3.2
- CursorBench 4.0
- cursorBench41
- DeepSWE
- FrontierCode 1.1 Main
- FrontierCode 2.0
- FrontierCode 1.1 Extended
- FrontierSWE v2
- LiveCodeBench
- LiveCodeBench Pro
- LiveCodeBench v6
- NL2Repo
- OpenHarmony Bench v1.2
- OpenHarmony Bench v1.0
- PostTrainBench
- PostTrainBench v1
- PostTrainBench v1.1
- ProgramBench
- React Native Evals
- ScarfBench
- SciCode
- SWE-Atlas Codebase QnA
- SWE-bench Pro
- SWE-bench Verified
- SWE-Rebench
- SWE-Marathon
- Vals LiveCodeBench
- Vals SWE-bench
- Vibe Code Bench
- VulcanBench Frontier v4
- VulcanBench v3
Math
- AIME 2024
- AIME 2025
- AIME26
- FrontierMath v2 (Tier 4)
- FrontierMath v2 (Tiers 1-3)
- HMMT Feb 2025
- HMMT Feb 2026
- HMMT Nov 2025
- IMOAnswerBench
- MATH-500
- MMAnswerBench
Knowledge & reasoning
- AA-GPQA Diamond
- HealthBench Professional (length-adjusted, Artificial Analysis)
- HealthBench Professional (raw, Artificial Analysis)
- AA-HLE
- AA MMLU-Pro
- ARC-AGI-2
- ARC-AGI-3 Provider Adapter
- ARC-AGI-1
- ARC-AGI-3
- Artificial Analysis Intelligence Index
- BBH
- BioBasicsBench
- BioMysteryBench (human-difficult)
- BioMysteryBench (human-solvable)
- C-Eval
- CritPt
- GPQA
- GPQA Diamond
- HealthBench (raw)
- HealthBench Hard
- HealthBench (length-adjusted)
- HealthBench Professional
- HealthBench Professional (raw)
- HLE
- HLE w/o tools
- HLE-Verified
- JevBench 1.4
- LABBench2
- MedXpertQA (Text)
- MMLU
- MMLU-Pro
- MMLU-Pro (Arcee)
- MMLU-Redux
- MMMLU
- AA-Omniscience Accuracy
- SimpleQA
- STEMBench
- SuperGPQA
- Vals GPQA Diamond
- Vals MMLU-Pro
Instruction following
Multilingual
Vision & documents
- AA-MMMU-Pro
- AI2D_TEST
- BabyVision
- BabyVision w/ Python
- BenchCAD Vision2Code (no tools)
- BenchCAD Vision2Code (tools)
- CC-OCR
- Chartography (no tools)
- Chartography (tools)
- CharXiv
- CharXiv w/o tools
- CountBench
- DynaMath
- ERQA
- LVBench
- MathVision
- MathVision w/ Python
- MedXpertQA (MM)
- MLVU (M-Avg)
- MMMU
- MMMU-Pro
- MMMU-Pro w/ Python
- MMVU
- OCRBench V2
- OfficeQA
- OfficeQA Pro
- OmniDocBench 1.5
- ParseBench-1
- ParseBench-2
- PerceptionBench
- RealWorldQA
- RefCOCO (avg)
- ScreenSpot Pro
- SimpleVQA
- SWE-bench Multimodal
- Video-MME (w/o subtitle)
- Video-MME (with subtitle)
- VideoMMMU
- V*
- We-Math
- ZeroBench
- ZeroBench w/ Python
Long context
General
- HLE-MultiChoice (Decision Index)
- Artificial Analysis Decision Index v1.0
- Artificial Analysis Decision Index
- When2Call (Decision Index)
- WildGuardTest (Decision Index)
- BabaIsBench
- BinaryAudit
- BlenderWorld (text-only)
- Chess Puzzles
- CompileBench
- Cookie-Bench
- DAYJOB
- DAYJOB Healthcare
- Decision Index Full Score
- EBR-bench
- ExploitBench
- ExploitBench Internal Port
- FACTS Multimodal
- FailureSensorIQ
- FrontierPhysics
- Furniture Assembly
- Decision Index Arts
- Decision Index Knowledge
- Decision Index Language
- Decision Index Retrieval
- Decision Index Tools
- MaxxBench
- MirrorCode
- Mystery Game Puzzles
- NYT Connections
- OTelBench
- SEC-Bench Pro
- SkillsBench official leaderboard (BenchFlow)
- SpireClimb
- StealthMark