Benchmark · math
GSM8K
GSM8K (Grade School Math 8K) is a benchmark of grade-school math word problems that each need several steps of arithmetic reasoning to solve; a model is scored on the percentage of problems where its final numeric answer is correct.
Read more
- Example
- A short word problem like "Natalia sold clips to 48 friends in April and half as many in May; how many clips did she sell in total?" — the model must work through the steps and give the final number.
- Scoring
- The score is accuracy — the share of problems whose final numeric answer exactly matches the reference answer; intermediate reasoning is not graded, only the final number.
- Verification
- Verification is automatic and exact-match: the model's final number is parsed out (typically after a delimiter like "####") and compared to the gold answer, so no human judgment is needed.
- Why it matters
- It became a standard, easy-to-check test of multi-step reasoning, and the well-known chain-of-thought prompting technique was shown to sharply improve results on it, making it a common yardstick for reasoning ability.
Worked example
Task
Maria has 3 boxes of crayons with 24 crayons in each box. She gives 15 crayons to her brother, then buys 2 more full boxes. How many crayons does she have now?
Solution
3 × 24 = 72; 72 − 15 = 57; 2 × 24 = 48; 57 + 48 = 105 → 105
Walkthrough
Multiply to get the starting count, subtract what she gives away, then add the newly bought boxes: 72 − 15 + 48 = 105. GSM8K grades only the final number extracted after the '####' delimiter by exact match, so the reasoning steps are not scored — only 105 counts.