Top-p (Nucleus Sampling)

Top-p sampling, also called nucleus sampling, is a decoding strategy that controls the randomness of a language model's output. At each step the model produces a probability for every possible next token; top-p keeps only the smallest set of tokens whose cumulative probability reaches a threshold p (for example 0.9 or 0.95), then samples the next token from that set. The long tail of unlikely tokens is never chosen at all, which suppresses incoherent derailments while still allowing genuine variety in the output.

Why it adapts better than top-k

The key advantage of top-p is that the size of the candidate pool — the "nucleus" — changes with the shape of the distribution. When the model is very confident, one or two tokens may already exceed the threshold, so the pool is tiny and the output is nearly deterministic; when the model is genuinely uncertain and probability is spread across many plausible continuations, the pool grows and permits more diversity. Contrast this with top-k sampling, which always keeps a fixed number of candidates regardless of context: k set low chokes creative passages, k set high lets junk tokens through in confident ones. Top-p's self-adjusting behavior is why it became a common default across local inference tools.

How it interacts with temperature

Top-p is almost always combined with a temperature setting, and the order of operations matters conceptually: temperature reshapes the probability distribution first — higher values flatten it, lower values sharpen it — and then top-p truncates the reshaped distribution before sampling. The two knobs answer different questions. Temperature asks "how bold should the model's preferences be?"; top-p asks "how much of the tail is even allowed on the table?" A low p produces tight, predictable answers well suited to factual or technical output; a higher p yields more varied, exploratory text suited to drafting and brainstorming. Some stacks add top-k as a hard cap alongside top-p, which mostly serves as a safety rail on unusual distributions.

Practical settings for self-hosters

Every serious local runtime — llama.cpp, Ollama, and the servers built on them — exposes these parameters directly, per request or in the model's configuration. That is one of the quiet perks of running your own models: hosted APIs decide which knobs you may touch, while your own stack gives you all of them. Sensible starting points are the defaults your model ships with; when tuning, move one parameter at a time and test against a fixed set of prompts so you can attribute the change. If outputs feel repetitive and flat, raise temperature or p slightly; if they wander or hallucinate structure, lower them. For pipelines that need reproducibility — evaluation harnesses, agent loops, anything you debug — pin the sampling settings and the random seed together, because "same prompt, different answer" is sampling working as designed, not a bug.

One member of a family

Nucleus sampling sits in a broader toolbox of decoding strategies alongside top-k, deterministic beam search, and newer schemes that adapt the truncation dynamically. For interactive chat and general text generation, though, temperature plus top-p remains the workhorse combination — simple, fast, and predictable enough to reason about.

A final note on defaults: model publishers usually ship recommended sampling settings for a reason — they tested them against the model's own quirks, and an unfamiliar model behaving badly is more often a sampling mismatch than a broken download. Before judging a new model, run it at its published settings; before shipping a pipeline, write the chosen values into the request explicitly rather than trusting whatever the runtime's current default happens to be. Sampling parameters are configuration, and configuration that lives in your own files, under version control, is one more small piece of the stack you actually control.