Greedy Decoding

Greedy decoding is the simplest text-generation strategy for a language model: at every step it selects the single token with the highest probability and never reconsiders. Because no randomness or search is involved, the output is fully deterministic for a given prompt and model, which makes it reproducible and cheap to compute. For a sovereign operator running a local model, greedy decoding is the predictable baseline against which every other sampling choice is measured — the "stock firmware settings" of text generation.

How it works

After each forward pass, the model produces a probability distribution over its vocabulary (the softmax of the logits). Greedy decoding simply takes the argmax — the single most likely token — appends it to the sequence, and repeats until an end-of-sequence token or length limit. It commits to the locally best choice at each position with no backtracking and no exploration. This is fast, adds zero sampling overhead per token during inference, and avoids the strange tangents that high-temperature sampling can produce, which is why it suits code completion, classification, extraction, and structured-output tasks where one correct answer is expected and creativity is a defect rather than a feature.

The trade-off: local optimum, global mediocrity

The weakness is myopia. The most probable next token does not always begin the most probable sequence: a token that looks best in isolation can lead down a path of lower overall quality, and greedy decoding has no mechanism to notice or recover. In practice greedy output tends to be flat, generic, and — the classic failure — prone to repetition loops, where the model falls into a cycle of phrases it cannot escape because each repeated token remains locally most likely. Two families of alternatives exist precisely to escape this trap: search methods like beam search, which keep multiple candidate sequences alive and compare their cumulative probabilities, and stochastic methods like temperature sampling, top-k, and top-p, which inject controlled randomness so the model can take locally suboptimal steps that pay off globally. Greedy decoding is the degenerate corner of both families: a beam search with beam width one, and a temperature sampler with temperature zero.

Using it well on a local stack

On a self-hosted setup running llama.cpp or Ollama, setting temperature to 0 gives you effectively greedy behaviour, and it is worth reaching for deliberately. Use it when you need repeatability: regression-testing a prompt after a model or quantization change, comparing two GGUF builds of the same model, extracting structured data where the same input must always yield the same output, or debugging a pipeline where sampling noise would mask the real problem. Determinism is a diagnostic tool — the same instinct as benchmarking a miner at fixed settings before tuning anything. One honest caveat: bit-exact reproducibility also depends on identical software builds, hardware, and batch behaviour, so "deterministic" means within a fixed stack, not across every machine you own. Then, for conversational or creative work, hand control back to a well-tuned sampler; the bland, loop-prone character of pure greedy output is the price of its predictability, and part of running your own models is knowing exactly when that price is worth paying.

A quick decision guide

Choosing a decoding strategy reduces to one question: does this task have one right answer? Extraction, format conversion, classification, deterministic test fixtures — temperature 0, greedy, done; any sampling here only adds noise you will later debug. Drafting, brainstorming, conversation — moderate temperature with top-p, because the local-optimum blandness of greedy output is most visible exactly where fluency matters. Code sits in between: greedy for completions and mechanical refactors, a touch of sampling when you want the model to propose alternatives worth comparing. When results disappoint, change one knob at a time and re-run the same prompt — the deterministic baseline is your control group, and keeping one is what separates tuning from thrashing. The habit is pure bench discipline, applied to tokens instead of voltages.