Softmax

Softmax is the function that turns a vector of arbitrary real-valued scores, called logits, into a probability distribution: every output is positive and the outputs sum to one. It works by exponentiating each score and dividing by the sum of all the exponentials, so the largest scores dominate while smaller ones still receive nonzero weight. That one-line recipe appears in two critical places inside every transformer language model, which makes softmax one of the most frequently executed operations in self-hosted inference, and the place where nearly all of the sampling controls you adjust actually take effect.

The name is a compression of soft maximum: where a hard max would assign everything to the single largest score and nothing to the rest, softmax distributes weight smoothly in proportion to the exponentiated scores, preserving a ranking while keeping every option alive. That softness is not a compromise but the point. It makes the function differentiable, which lets gradients flow through it during training, and it means the model's output is an honest statement of uncertainty rather than a single forced verdict. When a language model reports that two continuations are nearly equally likely, that information survives into sampling because softmax preserved it. The same property is what your sampling settings exploit: a distribution that keeps the runners-up alive is one you can flatten, sharpen, or truncate to taste.

Two jobs inside a transformer

First, attention uses softmax to convert query-key similarity scores into attention weights, the coefficients that decide how much each token blends in information from every other token in the context. The raw dot products can be any size; softmax normalizes them into a weighting that sums to one across the positions being attended to, with the position signal supplied by schemes like rotary position embedding. Second, the final output layer applies softmax across the entire vocabulary to produce the probability of each possible next token. Every word your local model generates is drawn from a distribution that softmax produced, and in sparse architectures the router that assigns tokens to experts in a Mixture of Experts model uses softmax as well.

Where your sampling knobs plug in

The controls exposed by every local inference stack operate on the logits just before or on the distribution just after softmax. Temperature rescales the logits before the exponential: low temperature sharpens the distribution toward the top choice, high temperature flattens it toward variety, and a temperature of zero collapses sampling into greedy decoding. Top-k and top-p sampling then truncate the resulting distribution, discarding the long tail of improbable tokens before the random draw. Understanding that all of these are just manipulations of one softmax output demystifies most of an inference server's settings page.

Numerical and practical notes

Naively exponentiating large logits overflows floating-point arithmetic, so every real implementation computes softmax in a numerically stable form by subtracting the maximum logit from all scores first, which changes nothing mathematically and everything practically. During training, softmax pairs naturally with cross-entropy loss, and the combination has a famously clean gradient, which is part of why the pairing is universal. For inference on modest hardware, it is worth knowing that attention's softmax is computed over every position in the context window, which is one reason long contexts cost memory and time, and why optimized kernels that fuse the softmax step deliver real speedups on consumer GPUs.

For the sovereign operator running models at home, softmax is the honest heart of the machine: the model's knowledge ends in a list of scores, softmax turns those scores into odds, and your sampling settings decide how boldly to roll the dice. See activation function for the other nonlinearity inside each layer, and temperature sampling for the knob most people reach for first.