Backpropagation

Backpropagation is the algorithm that makes training a neural network possible. After the network makes a prediction and a loss function measures how wrong it was, backpropagation works backward through the layers to compute the gradient — the partial derivative of the loss with respect to every weight in the model. Those gradients tell each weight which direction to move, and how strongly, to reduce the error. Popularized for neural networks by Rumelhart, Hinton, and Williams in 1986, it is essentially the chain rule of calculus applied systematically across a computation graph — an old piece of math, organized so well it trains models with billions of parameters.

Forward pass, backward pass

Training alternates two phases. In the forward pass, input flows through the network layer by layer to produce an output and a loss; the framework records the operations as it goes. In the backward pass, backpropagation propagates the error signal from the loss back toward the input, applying the chain rule at each recorded operation and reusing intermediate results so the entire gradient — every weight at once — is computed in a single sweep. That reuse is the whole trick: computed naively, per-weight derivatives would each require their own pass through the network, making large models untrainable in practice; backpropagation gets all of them for roughly the cost of one extra pass. The gradients are then handed to gradient descent (in practice, an optimizer like Adam), which actually updates the weights — backpropagation computes the map; the optimizer takes the step.

Why architecture is built around it

Backpropagation's failure modes shape network design. Because the backward pass multiplies many local derivatives together, the signal can shrink toward zero (vanishing gradients) or blow up (exploding gradients) across deep stacks — and a signal that dies before reaching the early layers means those layers never learn. This is exactly why deep models rely on residual connections, which give the gradient an additive shortcut past every block, and layer normalization, which keeps activations and gradients well-scaled. Careful weight initialization and gradient clipping serve the same master. A modern Transformer is, in large part, an architecture engineered so that backpropagation works reliably at extreme depth — that is what lets it learn from billions of tokens.

What it means for self-hosted AI

Backpropagation also explains the resource gap between training and inference that every self-hoster runs into. Inference is just the forward pass. Training must additionally store the intermediate activations from the forward pass (the backward pass needs them), compute gradients for every weight, and keep optimizer state on top — several times the memory of simply running the model. That is why a GPU that serves a model comfortably can be far too small to train it, and why techniques like LoRA-style parameter-efficient fine-tuning exist: they shrink the set of weights that need gradients, taming the backward pass enough to fit on hardware you own. Whenever you fine-tune a model on your own machine, the quiet workhorse spending most of those GPU cycles is this one 1986 algorithm, still running the show.

It is also worth knowing what backpropagation is not: it is not how the model runs, only how it learns. A trained network executing on your GPU performs no backward passes and stores no gradients, which is why inference-only hardware and heavily quantized weights work at all — the demanding, precision-sensitive part of the model's life ended at training time. Keeping the two phases distinct clarifies most hardware conversations in local AI: questions about running models are forward-pass questions, questions about customizing them are backward-pass questions, and the second class is always the expensive one.