Backpropagation

Backpropagation is the algorithm that makes training a neural network possible. After the network makes a prediction and a loss function measures how wrong it was, backpropagation works backward through the layers to compute the gradient, the partial derivative of the loss with respect to every weight in the model. Those gradients tell each weight which direction to move to reduce the error. Popularized for neural networks by Rumelhart, Hinton, and Williams in 1986, it is essentially the chain rule of calculus applied systematically across a computation graph.

Forward pass, backward pass

Training alternates two phases. In the forward pass, input flows through the network to produce an output and a loss. In the backward pass, backpropagation propagates the error signal from the output layer back to the input layer, reusing intermediate results so the whole gradient is computed in a single efficient sweep rather than perturbing each weight one at a time. The resulting gradients are then handed to gradient descent, which actually updates the weights.

Why architecture choices help it

Backpropagation struggles when gradients vanish or explode across many layers. This is exactly why deep models rely on residual connections and layer normalization to keep the backward signal well-scaled. Together these let a Transformer learn from billions of tokens.

See also gradient descent, the optimizer that consumes backpropagation's output.