Logit Bias

Logit bias is a per-token override applied during decoding that nudges or forbids specific tokens regardless of what the model would otherwise predict. It gives an operator surgical control over generation: ban a word entirely, force a particular answer token, or steer output away from a topic. Because it works at the lowest level of the sampling pipeline — on the raw scores themselves — it is one of the most precise levers available to someone self-hosting a model, and one of the few that carries a hard guarantee rather than a statistical tendency.

How it works

At every generation step the model emits a vector of logits, one raw score per vocabulary token, just before the softmax converts those scores into probabilities. A logit bias adds a fixed value to the scores of chosen token IDs at exactly that point. A large positive bias makes a token overwhelmingly likely to be selected; a large negative bias — effectively negative infinity in most implementations — removes the token from the vocabulary for that generation, since after softmax its probability is zero. The adjustment is applied at every step, so it persists across the entire sequence, and it composes with whatever sampling strategy is active: the biased logits flow into temperature scaling, top-k, and top-p filtering like any other scores.

The tokenizer gotcha

The sharpest edge in practice is that biases target token IDs, not words. A word you want banned may correspond to several distinct tokens: capitalized and lowercase forms, a variant with a leading space, pieces of the word split across subword boundaries, and its appearances inside longer words. Ban one token and the model cheerfully routes around it through another spelling. Doing logit bias properly means enumerating every relevant token ID with the exact tokenizer of the model being served — a small script, but a mandatory one. This is also why bias lists do not transfer between model families: different tokenizers, different IDs.

Practical uses and limits

Operators use logit bias to enforce a closed set of answers (biasing only the tokens for "yes" and "no" turns a chatty model into a reliable classifier), to hard-ban strings that must never appear in output, to discourage the end-of-sequence token when a model keeps stopping too early — or encourage it when the model rambles. In local pipelines it is a building block for lightweight guardrails that run entirely on your own hardware, with no cloud moderation layer in the loop. Its bluntness is also its limit: the bias is static and context-free, so forcing a token the context cannot support produces incoherent text, and steering broad style through token-level nudges rarely ends well. It is best reserved for narrow, well-understood constraints rather than general behaviour shaping.

Support in the self-hosted stack is broad: llama.cpp accepts per-token biases directly, and most OpenAI-compatible local servers expose the same logit_bias request field the cloud APIs standardized, taking a map of token IDs to bias values (commonly clamped to a range like −100 to 100, where the extremes act as ban and force). That API compatibility means a constraint developed against a local model travels unchanged if the backend swaps — provided the token IDs are regenerated for the new tokenizer. It is a small, sharp tool: a dozen lines of configuration that convert "the model usually behaves" into "this token cannot appear," which is exactly the kind of guarantee automation should be built on.

For schema-level control — constraining output to valid JSON or a formal grammar rather than pushing individual tokens — see grammar-constrained decoding, which generalizes the same masking idea. For the looping failure logit bias can help patch, see repetition penalty.