Layer Normalization

Layer normalization is a technique that stabilizes neural network training by re-scaling the activations inside each layer. Introduced by Ba, Kiros, and Hinton in 2016, it computes, for every token, the mean and standard deviation across that token's feature dimensions, normalizes the activations to zero mean and unit variance, and then applies a learned scale and shift so the network can still express whatever range it needs. It is one of those unglamorous plumbing components that quietly makes modern language models possible: without it, signals drifting through dozens of stacked layers grow or shrink until training falls apart.

Why not batch normalization

Batch normalization, the earlier standard from computer vision, normalizes each feature across all the examples in a training batch. That works poorly for language: sequences vary in length, batch statistics differ between training and inference, and small batches make the statistics noisy. Layer normalization sidesteps all of this by computing its statistics within a single token's own feature vector, independent of every other example. It therefore behaves identically during training and inference, at batch size one or one thousand — exactly what the variable-length sequences of a Transformer require, and exactly what you want when serving a model on your own hardware where batch sizes are small.

Where it sits in a Transformer

Every Transformer block pairs its attention and feed-forward sub-blocks with a normalization step and a residual connection. The placement matters more than it looks. Early designs applied normalization after each sub-block (post-norm); nearly all modern LLMs apply it before (pre-norm), which keeps the residual stream's scale stable and makes very deep stacks — dozens to over a hundred layers — trainable without delicate warm-up schedules. Without normalization somewhere in the loop, activation magnitudes compound layer over layer and gradient descent becomes unstable: gradients explode, vanish, or oscillate, and the loss curve turns into a seismograph.

RMSNorm, the common variant

Many current open-weight models replace classic layer norm with RMSNorm, a simplification that divides only by the root-mean-square of the activations and drops the mean-subtraction and bias terms. Empirically it trains just as well, and it is cheaper: fewer operations and fewer parameters per layer. On a datacenter GPU the saving is a rounding error; on a home rig running a quantized model through llama.cpp, thousands of normalization calls per generated token add up, and every shaved operation is tokens per second you keep. When you see RMSNorm listed in a model card, that is what it means — same stabilizing job, leaner execution. A final norm is also applied after the last Transformer block, just before the output projection, so normalization brackets the entire stack from first embedding to final logits.

Why a self-hoster should care

You will never tune a layer norm by hand, but the concept earns its place in a practical vocabulary for two reasons. First, normalization layers are famously sensitive to quantization: their statistics and learned scales often need to stay in higher precision even when the surrounding weights are compressed to four bits, which is why quantization formats treat them specially and why a badly quantized model can degrade in ways that trace back to its norms. Second, reading model architectures — pre-norm versus post-norm, LayerNorm versus RMSNorm — is part of evaluating what you are about to run on your own iron. The pattern echoes something familiar from the mining side of D-Central's world: stable systems are built by controlling drift at every stage, whether that is voltage regulation across a hashboard or activation scale across a hundred Transformer layers. See also backpropagation for how the gradients these norms stabilize actually flow.