Gradient Clipping

Gradient clipping constrains how large gradients can grow during backpropagation, guarding against the exploding-gradient problem. In deep networks, gradients are multiplied layer by layer as they flow backward; if their norms consistently exceed one, the product grows exponentially with depth and the resulting update can blow the model's weights to infinity or NaN in a single step. Clipping is the cheap, standard insurance against that failure — one line of configuration that turns a training-run-ending spike into a non-event.

Clipping by global norm

The most common form is clip-by-global-norm. The framework computes a single norm across all parameter gradients, and if that norm exceeds a chosen threshold, it scales every gradient down by the same ratio so the total norm equals the threshold. Because all gradients shrink proportionally, the direction of the combined update is preserved while only its length is capped. That direction encodes the steepest-descent path that backpropagation computed, which is why norm clipping is generally preferred over the older alternative of clipping each gradient value independently — value clipping distorts the update direction, effectively pointing the step somewhere the loss surface never asked for.

Where it matters

Gradient clipping is standard practice when training recurrent networks, long-sequence models, and transformers, where backpropagation through many steps or layers can produce sudden gradient spikes — often triggered by a rare batch, a sharp region of the loss surface, or an unlucky interaction with the learning-rate schedule. A typical threshold is 1.0, applied after gradients are computed but before the optimizer step of gradient descent. Frameworks expose it directly, for example as a max-grad-norm parameter, so it costs almost nothing to enable — and virtually every published large-model training recipe enables it.

Interactions with the rest of the training stack

Order of operations matters once other memory and precision tricks are in play. With gradient accumulation, clipping should apply to the fully accumulated gradient, not each micro-batch's contribution, or the effective threshold silently changes with the accumulation count. With mixed-precision training, gradients must be unscaled — reversing the multiplier that loss scaling applied — before the clip, otherwise you are clipping scaled values against an unscaled threshold. Mainstream trainers handle both orderings correctly, but anyone wiring a custom loop for a fine-tuning run on their own hardware should verify them, because both bugs fail silently and just make training subtly worse.

The gradient norm as a diagnostic

Clipping is also not a substitute for fixing pathological training. If the clip is firing on most steps, something upstream is wrong — learning rate, data outliers, a bad initialization, or numerical issues in a custom layer — and the clip is merely hiding the symptom while the run limps. Treat the mechanism like a circuit breaker on the bench: essential protection against rare faults, and a diagnostic prompt, never a component you design to run tripped.

A practical habit from the training bench: log the pre-clip gradient norm every step. It is one of the most informative single numbers in a run. A healthy run shows a norm that settles into a band; occasional spikes that trip the clip are normal and are exactly what the mechanism exists for. A norm that is always at the clip threshold means the threshold is doing the optimizer's job and the learning rate is likely too high; a norm collapsing toward zero signals the opposite problem, vanishing gradients. Clipping addresses gradients that grow too large; the counterpart hazard of gradients shrinking below numerical precision is handled by loss scaling in low-precision training. Both are routine safeguards layered on top of an optimizer state that already smooths updates over time.