Coding

Vals SWE-bench leaderboard

Name: benchgap: LLM benchmark scores, measured and estimated
Creator: benchgap

As of 2026-10-07, the highest measured score on Vals SWE-bench is 97.0% by Claude Opus 5. 150 more models have estimated scores, calibrated from the benchmarks they were measured on.

Measured scores: benchlm.ai.

#	Model	Score	Source
1	Claude Fable 5.1	100.0%	estimated ± 4.5 pp, medium confidence
2	Gemini 4 Argon	100.0%	estimated ± 4.5 pp, medium confidence
3	Claude Mythos 5	97.7%	estimated ± 6.8 pp, medium confidence
4	DeepSeek V4.1 Flash	97.3%	estimated ± 5.4 pp, low confidence
5	GPT-6 Astra	97.1%	estimated ± 1.4 pp, low confidence
6	Claude Opus 5	97.0%	measured
7	Claude Opus 5.5	96.9%	estimated ± 1.4 pp, medium confidence
8	DeepSeek V4 Pro 0813	96.4%	measured
9	GPT-5.6 Sol	96.2%	measured
10	Grok 4.6	95.6%	measured
11	Claude Sonnet 5.5	95.4%	estimated ± 1.4 pp, medium confidence
12	GLM-5.3	95.4%	measured
13	GPT-5.6 Terra	95.4%	measured
14	Claude Fable 5	95.0%	measured
15	Kimi K3	93.4%	measured
16	Ember-1	93.1%	estimated ± 6.8 pp, medium confidence
17	GPT-5.6 Luna	93.0%	measured
18	MiMo-V2.6-Pro	93.0%	estimated ± 7.6 pp, low confidence
19	Ornith-1.5-397B	92.6%	estimated ± 5.4 pp, medium confidence
20	Grok 4.7	92.4%	estimated ± 4.9 pp, high confidence
21	Hy4 preview	92.1%	estimated ± 5.4 pp, medium confidence
22	GLM-5.3-Flash	92.0%	measured
23	SWE-2	90.9%	estimated ± 2.8 pp, high confidence
24	Muse Spark 1.3	89.3%	estimated ± 5.4 pp, medium confidence
25	GPT-6 Sol	88.8%	estimated ± 7.6 pp, low confidence
26	DeepSeek V4 Flash 0731	88.8%	measured
27	Claude Opus 4.8	88.6%	measured
28	Composer 2	88.1%	estimated ± 1.4 pp, low confidence
29	Claude Opus 4.7 (Adaptive)	87.7%	estimated ± 5.4 pp, medium confidence
30	Composer 2 Fast	87.5%	estimated ± 1.4 pp, low confidence
31	Grok 4.5	86.6%	measured
32	Muse Spark 1.2	86.6%	measured
33	Claude Haiku 5.5	86.6%	estimated ± 2.8 pp, high confidence
34	Qwen3.8 Max Preview	86.4%	estimated ± 5.4 pp, medium confidence
35	Qwen3.8-27B	86.0%	measured
36	Qwen3.8 Max	85.6%	measured
37	GPT-6 Luna	85.0%	estimated ± 7.6 pp, low confidence
38	GPT-6.1 Sol	84.5%	estimated ± 7.6 pp, low confidence
39	Ling 3.1 Flash	84.4%	estimated ± 7.6 pp, low confidence
40	dots3-note Preview	83.8%	estimated ± 5.4 pp, medium confidence
41	Qwen3.8-Omni-Flash	82.9%	estimated ± 5.4 pp, medium confidence
42	GLM-5.2	82.8%	measured
43	GPT-5.5	82.6%	measured
44	SWE-1.7	82.3%	estimated ± 2.8 pp, high confidence
45	GPT-5.4	82.3%	estimated ± 1.4 pp, low confidence
46	Ornith-1.0-397B	82.2%	estimated ± 5.4 pp, medium confidence
47	Inkling-Small	82.2%	measured
48	Mistral Large 4	82.1%	estimated ± 6.3 pp, low confidence
49	Qwen3.8-Flash-Next	82.1%	estimated ± 5.4 pp, medium confidence
50	Claude Opus 4.7	82.0%	measured
51	Muse Spark 1.1	82.0%	measured
52	Claude Opus 4.6	81.6%	estimated ± 1.4 pp, medium confidence
53	Atria Dawn Preview	81.4%	estimated ± 7.5 pp, medium confidence
54	Laguna S 2.1	81.2%	estimated ± 7.5 pp, medium confidence
55	Seed 2.1 Pro	81.0%	estimated ± 5.4 pp, medium confidence
56	MiMo-V2.6-Flash	80.9%	estimated ± 7.6 pp, low confidence
57	Gemini 3.7 Flash	80.8%	measured
58	GPT-5.2	80.2%	estimated ± 6.3 pp, medium confidence
59	Claude Opus 4.6 (Adaptive)	80.2%	estimated ± 6.3 pp, medium confidence
60	Ornith-1.5-35B-A3B	80.2%	estimated ± 5.4 pp, medium confidence
61	Gemini 3.8 Flash	80.0%	measured
62	Claude Sonnet 5	79.6%	measured
63	Composer 2.5	79.6%	measured
64	Gemini 3.6 Flash	79.6%	measured
65	Gemini 3.1 Pro	78.8%	measured
66	Gemini 3.5 Flash	78.8%	measured
67	Quasar 438B	78.5%	estimated ± 5.4 pp, medium confidence
68	Apodex 1.1	78.2%	estimated ± 5.4 pp, medium confidence
69	Apodex 1.1 Mini	78.2%	estimated ± 5.4 pp, medium confidence
70	Kimi K2.7 Code	78.2%	measured
71	GPT-5.3 Codex	78.0%	measured
72	Sakana Fugu	78.0%	estimated ± 5.3 pp, low confidence
73	Step 5 Preview	77.8%	estimated ± 5.3 pp, low confidence
74	Sakana Fugu-Ultra	77.7%	estimated ± 5.3 pp, low confidence
75	Inkling	77.6%	measured
76	Seed 2.1 Turbo	77.5%	estimated ± 5.4 pp, medium confidence
77	Kimi K2.5	77.5%	estimated ± 1.4 pp, medium confidence
78	Claude Sonnet 4.6	77.4%	measured
79	Claude Opus 4.5	77.0%	estimated ± 5.4 pp, medium confidence
80	Hy3	76.8%	estimated ± 5.4 pp, medium confidence
81	GLM-5.1	76.4%	measured
82	Gemma 4 31B	76.2%	estimated ± 1.4 pp, medium confidence
83	Kimi K2.6	76.2%	measured
84	GLM-5	76.0%	estimated ± 1.4 pp, medium confidence
85	Qwen3.7 Plus	75.4%	estimated ± 5.3 pp, low confidence
86	Gemini 3.5 Flash-Lite	75.0%	measured
87	Gemini 3 Flash	75.0%	measured
88	MiniMax M3	75.0%	measured
89	Beam	74.6%	estimated ± 5.3 pp, low confidence
90	Grok 4	74.6%	estimated ± 1.4 pp, medium confidence
91	Muse Spark	74.4%	measured
92	BTL-4	74.2%	estimated ± 6.8 pp, medium confidence
93	MiMo-V2.5-Pro	74.0%	measured
94	GPT-OSS 120B	73.9%	estimated ± 1.4 pp, medium confidence
95	DeepSeek V3.2	73.9%	estimated ± 1.4 pp, medium confidence
96	MiniMax M2.7	73.8%	measured
97	MiMo-V2-Pro	73.6%	estimated ± 6.8 pp, medium confidence
98	GLM-5 (Reasoning)	73.6%	estimated ± 6.3 pp, medium confidence
99	GPT-OSS 20B	73.5%	estimated ± 1.4 pp, low confidence
100	Qwen3.6 Plus	73.4%	measured
101	Claude Sonnet 4.5 Thinking	73.3%	estimated ± 6.3 pp, medium confidence
102	GPT-5.1-Codex-Max	73.2%	estimated ± 6.3 pp, medium confidence
103	GPT-5.4 mini	73.0%	measured
104	Qwen 3.6 Max (preview)	72.8%	measured
105	Claude Opus 4.5 Thinking	72.6%	estimated ± 6.3 pp, medium confidence
106	Claude Sonnet 4.5	72.5%	estimated ± 6.8 pp, medium confidence
107	GPT-5.2-Codex	72.4%	measured
108	Grok 4.20	72.2%	measured
109	Ling 3.0 Flash VL	71.9%	estimated ± 7.6 pp, low confidence
110	Grok 4.3	71.4%	measured
111	Qwen3.5 397B	71.1%	estimated ± 6.8 pp, medium confidence
112	MiMo-V2.5	71.0%	measured
113	Qwen3.5 Plus	70.5%	estimated ± 6.3 pp, medium confidence
114	MiMo-V2-Flash	70.2%	estimated ± 5.4 pp, medium confidence
115	MiniMax M2.5	70.1%	estimated ± 6.3 pp, medium confidence
116	Qwen3.6-27B	70.0%	measured
117	Gemini 3 Pro	69.9%	estimated ± 6.3 pp, medium confidence
118	GPT-5.1	69.8%	estimated ± 5.4 pp, medium confidence
119	GPT-5 mini	69.8%	estimated ± 6.3 pp, medium confidence
120	GPT-5.4 nano	69.8%	measured
121	GLM-4.7	69.4%	measured
122	GPT-5.1-Codex	69.3%	estimated ± 6.3 pp, medium confidence
123	Muse Glimmer 30B	69.3%	estimated ± 5.3 pp, low confidence
124	MiMo-V2-Omni	69.2%	estimated ± 6.8 pp, medium confidence
125	Nemotron 3 Ultra	69.0%	measured
126	Qwen3.7 Max	68.8%	measured
127	Claude 4.1 Opus	68.8%	estimated ± 6.8 pp, medium confidence
128	Claude Haiku 4.5 Thinking	68.5%	estimated ± 6.3 pp, medium confidence
129	Kimi K2.5 (Reasoning)	67.9%	estimated ± 5.4 pp, medium confidence
130	MAI-Thinking-1	67.4%	estimated ± 6.8 pp, medium confidence
131	Qwen3.5-122B-A10B	67.1%	estimated ± 5.4 pp, medium confidence
132	Ornith-1.0-35B	66.8%	estimated ± 5.4 pp, low confidence
133	Claude Haiku 4.5	66.6%	measured
134	Mistral Medium 3.5 128B	66.4%	measured
135	Claude 4 Sonnet	66.3%	estimated ± 6.8 pp, medium confidence
136	MAI-Code-1.1-Flash	66.2%	estimated ± 6.8 pp, medium confidence
137	Qwen3.5-27B	65.9%	estimated ± 6.8 pp, medium confidence
138	Hy3 Preview	65.9%	estimated ± 5.3 pp, low confidence
139	A.X K2	65.5%	estimated ± 5.3 pp, low confidence
140	DeepSeek V3 0324	65.4%	estimated ± 7.6 pp, low confidence
141	Ling 3.0 Flash	65.2%	measured
142	North Mini Code	65.1%	estimated ± 7.6 pp, low confidence
143	DeepSeek V3.2 (Thinking)	65.0%	estimated ± 6.3 pp, medium confidence
144	Ling 3.0 Flash FP8	64.4%	estimated ± 5.3 pp, low confidence
145	Qwen3.5 Flash	64.4%	measured
146	Qwen3 Max	64.1%	estimated ± 6.3 pp, medium confidence
147	Ornith-1.5-9B	63.9%	estimated ± 5.4 pp, low confidence
148	Laguna XS 2.1	63.8%	estimated ± 6.8 pp, medium confidence
149	GLM-4.6	63.8%	estimated ± 6.3 pp, medium confidence
150	Grok Code Fast 1	63.7%	estimated ± 6.8 pp, medium confidence
151	Solar Pro 4	63.4%	estimated ± 6.8 pp, medium confidence
152	Solar Open 2	63.2%	estimated ± 6.8 pp, medium confidence
153	Gemini 3.1 Flash-Lite	62.8%	measured
154	o1	62.7%	estimated ± 5.4 pp, medium confidence
155	Grok 4.1 Fast (Reasoning)	62.6%	estimated ± 6.3 pp, medium confidence
156	Step 3.7 Flash	62.6%	estimated ± 5.4 pp, medium confidence
157	Gemma 4 26B A4B	62.4%	estimated ± 5.4 pp, medium confidence
158	Grok 4 Fast (Reasoning)	61.8%	estimated ± 6.3 pp, medium confidence
159	Qwen3.5-35B-A3B	61.5%	estimated ± 6.8 pp, medium confidence
160	GPT-5 (high)	61.3%	estimated ± 5.4 pp, medium confidence
161	Nemotron 3 Super 100B	61.2%	estimated ± 5.4 pp, medium confidence
162	Granite 4.2 30B	61.2%	estimated ± 5.3 pp, low confidence
163	LongCat-Flash-Lite-Sparse	60.1%	estimated ± 6.8 pp, medium confidence
164	Qwen3.6-35B-A3B	59.7%	estimated ± 5.4 pp, low confidence
165	Mercury 2.5	59.4%	estimated ± 5.3 pp, low confidence
166	o1-preview	58.5%	estimated ± 5.4 pp, medium confidence
167	K-EXAONE 2.0	57.9%	estimated ± 5.3 pp, low confidence
168	Laguna M.1	57.6%	measured
169	K-Exaone	57.1%	estimated ± 5.4 pp, low confidence
170	Ornith-1.0-9B	56.5%	estimated ± 5.4 pp, low confidence
171	Gemma 4 12B	56.2%	estimated ± 5.4 pp, low confidence
172	Laguna XS.2	55.2%	measured
173	Granite 4.2 8B	54.4%	estimated ± 5.3 pp, low confidence
174	Gemini 2.5 Pro	54.4%	measured
175	Command A+	53.9%	estimated ± 5.4 pp, low confidence
176	Mistral Small 4	53.0%	estimated ± 5.4 pp, low confidence
177	Mistral Small 4 (Reasoning)	53.0%	estimated ± 5.4 pp, low confidence
178	Trinity-Large-Preview	52.4%	estimated ± 5.4 pp, low confidence
179	Trinity-Large-Thinking	52.4%	estimated ± 5.4 pp, low confidence
180	Gemini 1.5 Pro	50.8%	estimated ± 5.4 pp, low confidence
181	DeepSeek V3	50.4%	estimated ± 5.4 pp, low confidence
182	Ternary Bonsai 2 27B	50.0%	estimated ± 6.8 pp, low confidence
183	GPT-4 Turbo	49.2%	estimated ± 5.4 pp, low confidence
184	Solar Pro 3	48.4%	estimated ± 7.6 pp, low confidence
185	GPT-4.1 mini	48.3%	estimated ± 5.4 pp, low confidence
186	Mistral Large 3	48.2%	estimated ± 5.4 pp, low confidence
187	Claude 3 Opus	47.8%	estimated ± 5.4 pp, low confidence
188	Ling 3.0 Tiny	46.7%	estimated ± 7.6 pp, low confidence
189	Llama 4 Maverick	45.4%	estimated ± 5.4 pp, low confidence
190	Celeris-1	44.0%	estimated ± 5.4 pp, low confidence
191	Nemotron 3 Nano 30B	44.0%	estimated ± 5.4 pp, low confidence
192	Ultravox v0.6 Llama 3.3 70B	42.2%	estimated ± 5.4 pp, low confidence
193	GPT-4o mini	41.8%	estimated ± 5.4 pp, low confidence
194	GPT-4.1 nano	41.6%	estimated ± 5.4 pp, low confidence
195	GPT-4.1	41.4%	estimated ± 6.8 pp, low confidence
196	Gemma 3 27B	40.8%	estimated ± 5.4 pp, low confidence
197	Nemotron 3 Nano Omni 30B A3B	40.7%	estimated ± 5.3 pp, low confidence
198	Gemma 4 E4B	40.3%	estimated ± 5.4 pp, low confidence
199	ZAYA1-74B-Preview	39.5%	estimated ± 6.8 pp, low confidence
200	Llama 4 Scout	39.4%	estimated ± 5.4 pp, low confidence
201	LFM2.5-2.6B	39.1%	estimated ± 5.4 pp, low confidence
202	Gemma 4 E2B	38.7%	estimated ± 5.4 pp, low confidence
203	Nemotron 3.5 Lightning 30B A3B NVFP4	38.4%	estimated ± 5.3 pp, low confidence
204	o3-mini	34.1%	estimated ± 6.8 pp, low confidence
205	LLaDA2.2-flash	34.1%	estimated ± 6.8 pp, low confidence
206	Claude 3.5 Sonnet	33.7%	estimated ± 6.8 pp, low confidence
207	Agents-A1-4B	31.6%	estimated ± 5.3 pp, low confidence
208	Ling 2.6 Flash	21.8%	estimated ± 5.3 pp, low confidence
209	MiniCPM5-2B	19.4%	estimated ± 5.3 pp, low confidence
210	Granite 4.2 3B	12.8%	estimated ± 5.3 pp, low confidence

Vals SWE-bench leaderboard

All leaderboards

Agentic · terminal

Agentic · tools

Coding

Math

Knowledge & reasoning

Instruction following

Multilingual

Vision & documents

Long context

General