Decoding Strategy

A decoding strategy is the algorithm that converts a language model's raw per-step probability distribution into a concrete sequence of output tokens. The model itself only produces a distribution over the vocabulary at each position; the decoding strategy decides how that distribution is turned into text. For a sovereign operator running a model locally, the decoding strategy is often the single biggest lever over output quality that requires no retraining — only configuration you control on your own hardware.

Deterministic search

The simplest family always takes the model's word for it. Greedy decoding picks the single most probable token at every step; beam search keeps several candidate sequences alive and picks the most probable overall path. Both are reproducible — the same prompt yields the same output — which is exactly what you want for extraction, classification, and code completion. The cost is blandness: deterministic methods gravitate toward safe, repetitive phrasing, and beam search in particular tends to produce short, generic completions on open-ended prompts.

Stochastic sampling

Sampling methods draw from the distribution instead of maximising it, with a handful of knobs shaping the draw. Temperature rescales the distribution: below 1.0 it sharpens toward the top choices, above 1.0 it flattens toward randomness. Top-k truncates the candidate pool to the k most probable tokens, while top-p (nucleus) sampling keeps the smallest set of tokens whose probabilities sum to p, adapting the pool size to the model's confidence. Repetition and frequency penalties push back against loops. These controls compose: a common recipe is nucleus sampling at moderate temperature with a light repetition penalty, tuned per task and per model.

Constraints and acceleration

Two further families change the game without changing the vibe. Grammar-constrained decoding masks out every token that would violate a required format — JSON, a fixed schema, a command syntax — so the output is machine-parseable by construction, which matters enormously when a local LLM drives tools or APIs rather than a human reader. Speculative decoding, by contrast, changes how tokens are computed, not which ones are chosen: a small draft model proposes several tokens and the large model verifies them in one pass, often doubling throughput on the same GPU with identical output. On self-hosted inference servers these features are configuration flags, not vendor privileges.

Choosing a strategy

There is no universally best strategy; the right choice tracks the task. Reliability-critical work — parsing an error log, filling a template, answering from retrieved documents — favors greedy or low-temperature settings. Creative drafting favors higher temperature with nucleus sampling. Machine-to-machine output favors grammar constraints. And every choice interacts with the tokenizer and context window you are working within: a strategy tuned for one model rarely transfers unchanged to another. The craftsman's approach is to fix a small evaluation set, sweep the two or three parameters that matter, and write down what won — the same discipline you would apply to tuning a miner, applied to tuning a model.

In practice a decoding configuration is a handful of numbers — temperature, top-p or top-k, repetition penalty, maximum tokens, stop sequences — and treating them as part of the deliverable pays off. The same open-weight model can behave like two different products under two different configurations, so when a local pipeline works, record the model version and the sampler settings together; when it misbehaves, suspect the sampler before blaming the weights. Common symptoms map cleanly: word-salad output usually means temperature or top-p set too loose, endless repetition means a missing penalty or greedy decoding on a weak model, truncated answers mean a token limit, and malformed JSON means you needed constrained decoding rather than politer prompting. Debugging the decode layer first is the cheapest fix in local AI — no downloads, no retraining, just turning the right knob.