Constrained Decoding

Constrained decoding, also called guided or structured decoding, is a method that guarantees a language model's output conforms to a predefined format such as valid JSON, a regular expression, or a context-free grammar. Instead of generating freely and hoping the result parses, the decoder restricts the model at every step so that only tokens which keep the output legal can be chosen. This turns “please respond in JSON” from a polite request that sometimes fails into a hard guarantee the format will hold.

How the constraint is enforced

The target format is compiled into a state machine — typically a finite-state machine for regular grammars or a pushdown automaton for nested structures like JSON. At each generation step the automaton's current state defines exactly which next tokens are valid. The decoder applies a mask to the model's output probabilities, setting the probability of every illegal token to zero, then samples only from the survivors. Libraries such as Outlines and XGrammar precompute these state-to-token maps so the masking adds little overhead, and in some cases throughput actually rises because the model never wastes tokens on malformed output that would have to be discarded and generated again from scratch.

Why sovereign operators care

When a locally hosted model feeds a downstream program — an automation script, a database write, or a tool call — malformed output breaks the pipeline. Constrained decoding makes the model a reliable component: the output is always parseable, so no brittle text-scraping or repair logic is needed to clean up after it. It is the foundation of dependable function calling and agent workflows running entirely on one's own hardware, where there is no vendor API to lean on and no support desk when a script silently mangles a response. For anyone wiring a home model into real automation, it converts a probabilistic tool into a dependable one.

The limits worth knowing

Constraints govern shape, not truth. A model forced to emit valid JSON will always emit valid JSON, but it can still put the wrong values in the fields, so structure is a guarantee of form and never of correctness. Overly rigid grammars can also steer a model away from its best answer by forbidding a token it would have chosen next, occasionally degrading quality in exchange for structure. The format should therefore encode only what the downstream consumer genuinely requires and leave the model free everywhere else. Used with that restraint, it is one of the highest-leverage reliability tools in local serving.

A composable technique

One reason constrained decoding is so widely adopted is that it is orthogonal to almost everything else. It touches only the final probability distribution, after the model has done its work, so it can be layered on top of any model and any serving stack without retraining or fine-tuning.

A practical wrinkle worth knowing is tokenization. Grammars are usually written in terms of characters or bytes, but a language model emits tokens, and a single token can straddle a grammar boundary — for example a token that contains both the end of a string and the following comma. Robust libraries handle this by reasoning at the token level, precomputing which tokens are legal in each state rather than assuming a clean one-character-per-step model. Getting that mapping right is what separates a constraint engine that is genuinely reliable from one that occasionally paints itself into a corner where no legal token exists. For a self-hoster the lesson is to lean on well-tested libraries for anything beyond the simplest formats, because the edge cases are subtle and a hand-rolled masker is easy to get almost right and therefore quietly wrong.

Constrained decoding operates purely on the model's output logits at generation time, so it composes cleanly with throughput techniques like continuous batching, the standard KV cache, and the latency smoothing of chunked prefill.