Grammar-Constrained Decoding

Grammar-constrained decoding restricts a language model's next-token choice to only those tokens that keep the partial output valid against a formal grammar, schema, or regular expression. It is the mechanism behind reliable structured output: instead of hoping the model produces well-formed JSON or a valid command, you make it impossible for it to do otherwise. For a sovereign operator wiring a local model into tooling, this turns a probabilistic generator into something closer to a parser-safe interface.

How it works

At every decoding step the constraint engine evaluates which tokens the grammar permits given the text generated so far. Tokens that would make the output invalid have their logits set to negative infinity, so after the softmax their probability is zero and they can never be sampled. The model then picks from the remaining legal tokens using whatever sampling strategy is configured. A common grammar format is GBNF (GGML Backus-Naur Form), used by llama.cpp, which can be generated automatically from a JSON schema.

Why it matters

Without constraints, even capable models occasionally emit a trailing comma, an unescaped quote, or a hallucinated field, and a single bad character can crash a downstream parser. Grammar constraints eliminate that class of failure entirely at the generation level. The cost is some implementation complexity and the need to express your requirements as a grammar; overly rigid grammars can also occasionally hurt reasoning quality.

This technique underpins JSON Mode and pairs naturally with token-level controls like Logit Bias.