Overview

OpenAI 发布的 8500 道小学数学应用题，需要 2-8 步推理求解，评估模型的多步数学推理与基础算术能力。

Metrics

Metric	Unit	Direction
accuracy	%	↑ Higher is better

Sources

Model Score Ranking

#	Model	Vendor	Score
1	Claude 3.5 Sonnet	anthropic	96.4
2	GPT-4o	openai	95.8
3	Gemini 1.5 Pro 002	google	95
4	Claude 3.5 Sonnet (2024-10-22)	anthropic	94
5	GPT-4o (2024-08-06)	openai	93.3
6	Grok-2	xai	93.2
7	Mistral Large 2	mistral	93
8	Gemini 2.0 Flash	google	92.7
9	GPT-4o (2024-05-13)	openai	92.6
10	o1 Preview	openai	92.5
11	Claude 3 Opus	anthropic	91.8
12	Gemini 1.5 Pro	google	91.7
13	GPT-4	openai	91.7
14	Gemini 2.0 Flash Thinking	google	91.3
15	Sonar Huge	other	90.4
16	Yi Vision	other	89.9
17	Sonar Reasoning	other	89.8
18	DeepSeek V3	deepseek	89.3
19	Llama 3.1 405B	meta	89.2
20	Hermes 3 Llama 3.1 405B	other	88.9
21	o1	openai	88.9
22	Qwen2.5 72B	alibaba	88.4
23	Command Nightly	cohere	87.2
24	Qwen2 72B	alibaba	87
25	Command R	cohere	86.9
26	Llama 3.3 70B	meta	86.9
27	o1 mini	openai	86.6
28	Gemini 1.5 Flash-8B 002	google	86.3
29	Gemini 1.0 Flash	google	86.2
30	Gemini 1.0 Ultra	google	85.9
31	GPT-4 1106 Preview	openai	85.4
32	GPT-4 Turbo	openai	85.2
33	Llemma 7B	other	84.9
34	Llama 3.1 Nemotron 70B	meta	84.6
35	GPT-4 Vision Preview	openai	84.5
36	Sonar Large	other	84.5
37	Orca 2 13B	other	84.1
38	Claude 3 Sonnet	anthropic	84
39	Qwen1.5 14B	alibaba	84
40	Qwen2.5 14B	alibaba	84
41	Command R (08-2024)	cohere	83.7
42	Microsoft WizardLM 2 8x22B	other	83.7
43	Microsoft WizardMath 7B v1	other	83.7
44	GPT-4 Vision	openai	83.6
45	Grok-2 Mini	xai	83.6
46	Gemini 1.5 Flash-8B	google	83.5
47	Jamba 1.5 Large	other	83.5
48	Grok-2 Vision	xai	83.4
49	Qwen2 57B	alibaba	83.3
50	Sonar Small	other	83.2
51	DeepSeek Math 7B	deepseek	83.1
52	GPT-4 0125 Preview	openai	82.9
53	Nous Hermes 2 Yi 34B	other	82.7
54	Claude 3 Sonnet (2024-02-29)	anthropic	82.6
55	Llama 3.2 90B Vision	meta	82.6
56	Yi 1.5 34B	other	82.6
57	Mistral Medium	mistral	82.5
58	Nous Hermes 2 Mixtral 8x7B	other	82.5
59	Gemini 1.0 Pro	google	82.4
60	Claude 3 Opus (2024-02-29)	anthropic	82.3
61	GLM-4 Air	other	82.3
62	Claude 3 Haiku	anthropic	82.2
63	Jamba Instruct	other	82.2
64	GLM-4 Plus	other	81.8
65	GPT-4 32K	openai	81.6
66	Yi Large	other	81.3
67	Qwen1.5 32B	alibaba	81.1
68	Yi Large Turbo	other	81
69	Mistral Large	mistral	80.9
70	Qwen1.5 72B	alibaba	80.9
71	Command R+ (08-2024)	cohere	80.8
72	Qwen1.5 110B	alibaba	80.6
73	Zephyr ORPO 141B Alpha	other	80.6
74	DeepSeek LLM 67B	deepseek	79.9
75	Jamba 1.5	other	79.8
76	Mixtral 8x7B	mistral	79.7
77	Mistral Small 3	mistral	79.6
78	Claude 3 Haiku (2024-03-07)	anthropic	79.5
79	StableLM 2 12B	other	79.1
80	DeepSeek V2	deepseek	78.8
81	Gemini 1.5 Flash 002	google	78.8
82	Llama 3.1 70B	meta	78.8
83	DBRX Instruct	other	78.7
84	Mixtral 8x22B	mistral	78.6
85	GLM-4 Flash	other	78.4
86	WizardLM Team WizardLM 2 8x22B	other	77.9
87	NVIDIA Llama 3.1 Nemotron 70B	other	77.8
88	Mistral Small	mistral	77.7
89	Falcon 180B	other	77.5
90	DeepSeek V2 Chat	deepseek	77.4
91	Gemma 2 27B	google	77.4
92	Hermes 3 Llama 3.1 70B	other	76.4
93	Qwen2.5 32B	alibaba	76.3
94	Llama 3 70B	meta	76.2
95	Jamba 1.5 Mini	other	75.7
96	Claude 3.5 Haiku	anthropic	75.5
97	Phi-3.5 MoE	other	75.5
98	DBRX Base	other	75.4
99	Gemini 1.5 Flash	google	75.2
100	GPT-4o mini	openai	75.1
101	GLM-4V 9B	other	73.5
102	Qwen2 7B	alibaba	73.2
103	Hermes 3 Llama 3.1 8B	other	72.9
104	Yi 1.5 9B	other	72.8
105	Mathstral 7B	mistral	72.4
106	OLMo 7B	other	71.3
107	Phi-4	other	71.1
108	Yi 1.5 6B	other	70.4
109	Mistral Nemo	mistral	70
110	Command R7B	cohere	69.7
111	DeepSeek Coder 33B	deepseek	69.7
112	StarCoder2 15B	other	69.6
113	Mistral 7B v0.3	mistral	69.4
114	StableCode 3B	other	68.5
115	Command R+	cohere	68.4
116	Llama 3.2 11B Vision	meta	68.2
117	Codestral Mamba	mistral	67.8
118	StarCoder2 7B	other	65
119	Gemma 2 9B	google	64.3
120	Qwen2.5 7B	alibaba	63.7
121	Gemma 7B	google	63.6
122	OLMo 1.7 7B	other	63.5
123	Codestral	mistral	63.3
124	Mistral 7B v0.1	mistral	63.1
125	DeepSeek Coder V2	deepseek	62.9
126	OLMo 7B SFT	other	62.8
127	Phi-3 Small	other	62.7
128	Falcon 40B	other	61.3
129	Code Llama 70B	meta	61
130	Code Llama 34B	meta	59.8
131	MPT 30B	other	59.6
132	Nous Hermes 2 Solar 10.7B	other	59.6
133	Phi-3 Medium	other	59
134	Argilla Notus 7B v1	other	58.8
135	Llama 3.1 8B	meta	58.8
136	OLMo 7B Instruct	other	58.7
137	Microsoft WizardLM 2 7B	other	58.5
138	GLM-4 9B Chat	other	58
139	Mistral Tiny	mistral	57.2
140	OLMo 2 1124 7B	other	57.2
141	GPT-3.5	openai	56.6
142	Llama 3 8B	meta	56.6
143	Mistral 7B v0.2	mistral	56.2
144	Code Llama 13B	meta	55.9
145	Code Bison	google	55
146	OpenChat 3.5 1210	other	55
147	DeepSeek Coder 7B	deepseek	54.7
148	GPT-3.5 Turbo 16K	openai	52.8
149	Chat Bison	google	52.4
150	Code Llama 7B	meta	52.2
151	Llama 2 70B	meta	51.8
152	OpenChat 3.6 8B	other	51.8
153	Ministral 8B	mistral	51.7
154	Phi-1	other	51.6
155	TigerBot 70B Chat	other	51.1
156	StarChat2 15B v0.1	other	50.7
157	Zephyr 7B Beta	other	50.4
158	PaLM 2	google	49
159	Llama 3.2 1B	meta	48.9
160	Grok Beta	xai	48.7
161	Yi 34B	other	48.3
162	StarCoder2 3B	other	48.2
163	StableLM 3 4B	other	48.1
164	Microsoft WizardCoder Python 34B	other	46.7
165	Flan-T5 XL	other	46.5
166	Llama 3.2 3B	meta	46.4
167	Llama Guard 2 8B	meta	46.3
168	Claude 2.1	anthropic	45.7
169	MPT 7B	other	44.6
170	Open-Platypus	other	43.9
171	StableLM 2 1.6B	other	43.5
172	Text Bison	google	42.8
173	Baichuan2 13B Chat	other	41
174	Phi-3 Mini	other	40.9
175	Flan-UL2	other	40.6
176	Baichuan2 7B Chat	other	40
177	Command Light	cohere	39.8
178	ChatGLM3 6B	other	37.9
179	Yi 6B	other	37
180	Phi-2	other	36.8
181	Qwen2 1.5B	alibaba	36.2
182	Qwen2.5 0.5B	alibaba	36.2
183	StableLM Zephyr 3B	other	36.2
184	Flan-T5 XXL	other	35.2
185	Grok Vision Beta	xai	35.2
186	Llama 2 13B	meta	34.6
187	GPT-3.5 Turbo	openai	34.3
188	Phi-1.5	other	34.2
189	Claude Instant 1	anthropic	34
190	Jurassic-2 Mid	other	33.2
191	Qwen2.5 1.5B	alibaba	33.2
192	Zephyr 7B Alpha	other	32.8
193	Claude 2	anthropic	32.5
194	Ministral 3B	mistral	32.2
195	Nous Capybara 34B	other	31.9
196	Qwen2.5 3B	alibaba	31.3
197	Llama 2 7B	meta	31
198	Nous Capybara 7B	other	31
199	Llama Guard 3 8B	meta	30.9
200	Jurassic-2 Ultra	other	30.3
201	Phi-3 Vision	other	30
202	Capybara 1.5B	other	28.5
203	Gemma 2B	google	27.8
204	Phi-3.5 Mini	other	22.1
205	Embed English v3	cohere	0

GSM8K (Grade School Math 8K)

Descrizione

OpenAI 发布的 8500 道小学数学应用题，需要 2-8 步推理求解，评估模型的多步数学推理与基础算术能力。

Specifiche principali

Categoria	Licenza	Ultimo aggiornamento
math	MIT	2021-10-01

Benchmark

Unità
%

Fonti

URL ufficiale

https://github.com/openai/grade-school-math