GSM8K (Grade School Math Benchmark)

GSM8K (Grade School Math 8K) is a benchmark released by OpenAI in 2021 to measure multi-step mathematical reasoning in language models. It contains roughly 8,500 high-quality, linguistically diverse grade-school word problems written by human authors, split into about 7,500 training and 1,000 test problems. Each problem is solvable by a capable middle-school student but requires between two and eight sequential steps of basic arithmetic, so success depends on planning and executing a chain of operations rather than recalling a single fact. That design made it, for several years, the reference test for whether a model can actually reason through a problem instead of pattern-matching an answer.

What it measures

A model reads a short story problem — apples bought, distances driven, money shared — and must produce the correct final numerical answer, typically after working through intermediate steps. GSM8K is the benchmark that most clearly demonstrated the value of chain-of-thought prompting: encouraging a model to write out its reasoning before answering sharply improves accuracy, because each intermediate step becomes context that conditions the next. Scores are reported simply as the percentage of test problems answered correctly, and the dataset's careful human authorship keeps its estimated label-error rate below 2%, so a wrong answer almost always reflects a genuine model failure rather than a bad question.

Strengths and caveats

GSM8K's virtue is isolation: by keeping the arithmetic elementary, it separates reasoning ability from advanced mathematical knowledge, so a low score signals a reasoning failure rather than missing math facts. Its weaknesses are the standard fate of successful benchmarks. Frontier and even strong open models now score in the high nineties, compressing the useful signal at the top — a benchmark near saturation mostly distinguishes bad models from adequate ones. Worse, the problems have circulated publicly for years, so contamination (test problems leaking into training data) inflates scores in ways that are hard to detect from the outside. A contamination-controlled variant, GSM1k, was built with freshly written problems of matched difficulty partly to measure this effect, and it found meaningful gaps for some model families. Harder successors such as MATH push into competition-level problems where headroom remains.

Using it when choosing a local model

For a self-hoster picking an open model to run on their own hardware, GSM8K remains a useful floor check rather than a differentiator: a model that stumbles on grade-school word problems will not handle multi-step practical tasks — unit conversions, profitability arithmetic, configuration logic — reliably. It is most informative when read alongside complementary benchmarks and at the quantization level you actually plan to run, since aggressive quantization can shave a few points off reasoning-heavy tasks before it visibly degrades casual chat. Published scores almost always reflect full-precision weights, so treat them as an upper bound for your local deployment.

Reported scores also hide methodology choices worth checking before you compare models. Accuracy differs depending on whether the model answers in one shot or with several worked examples in the prompt, and self-consistency — sampling many reasoning chains and taking a majority vote on the final answer — can add several points over a single greedy pass at the cost of proportionally more compute. Leaderboard entries rarely use identical settings, so a two-point gap between models is usually noise while a twenty-point gap is signal. When in doubt, run a slice of the test set yourself against your own quantized deployment; the dataset is small enough that an evening of local compute settles the question.

GSM8K's reasoning focus complements the broad-knowledge MMLU benchmark and the science-reasoning GPQA benchmark; together the three give a self-hoster a quick triangulation of whether an open model can reason, recall, and resist shallow pattern-matching — not just chat fluently.