Residual Connection

A residual connection, also called a skip connection, is a shortcut that adds a layer's input directly to its output before passing the result onward — output equals input plus the layer's transformation of that input. Introduced in the 2015 ResNet paper by He and colleagues, it was the breakthrough that made networks hundreds of layers deep trainable. Instead of forcing each block to learn a complete transformation from scratch, the block only has to learn the residual — the difference from simply passing the input through unchanged — which turns out to be a far easier target.

Why deep models need it

Before residual connections, deeper networks paradoxically performed worse than shallower ones — not from overfitting, but because they became untrainable. The culprit is what happens during backpropagation: as the error signal is propagated backward through many layers, repeated multiplication tends to shrink it toward zero — the vanishing-gradient problem — so the early layers barely learn at all. A residual connection gives that signal a clean additive path straight through: because the shortcut contributes the identity to every layer's output, the gradient always has an unobstructed route to earlier layers, no matter how deep the stack. Depth stops being a liability. There is also a useful intuition on the forward side: with skip paths everywhere, each block makes an incremental refinement to a running representation rather than re-deriving everything, and a block that has nothing useful to add can safely learn to contribute approximately nothing.

In the Transformer block

Every sub-block in a Transformer — both the self-attention and the feed-forward network — is wrapped in a residual connection paired with layer normalization. The pattern repeats identically dozens of times up the stack: normalize, transform, add back the input. It is simple and non-negotiable — remove the residual paths and a deep Transformer fails to converge at all. Modern large language models are only possible because this one architectural trick lets gradient flow survive stacks of many dozens of layers; in a real sense, every model you run locally is a ResNet-era idea wearing an attention mechanism.

Why a self-hoster should care

Beyond intellectual hygiene, the residual structure has practical fingerprints in the tooling you use. Because each layer is an increment on a shared residual stream, techniques like layer skipping, early exit, and pruning of whole blocks can work at all — dropping a layer degrades a residual network gracefully instead of severing the signal path. The residual stream is also the natural place interpretability work reads and edits a model's intermediate state. When you watch fine-tuning converge smoothly on a deep model using modest hardware, residual connections are a large part of why the optimization is that well-behaved. For the bigger picture of how these pieces assemble into a working model, see the Transformer entry — the residual connection is the quiet plumbing that lets everything else in it stack.

The idea has also aged remarkably well. A decade after ResNet, essentially every serious architecture — vision, language, audio, multimodal — is organized around a residual stream, and proposed replacements keep rediscovering additive shortcuts under new names. When a single mechanism survives that many architectural revolutions unchanged, it usually means it solved the actual problem rather than a fashionable one; for residual connections, the actual problem was letting optimization signals travel arbitrary distances, and no deeper alternative has displaced the simple answer of adding the input back in. For a working intuition, picture the network as a conveyor belt carrying a representation forward while each station adds a small correction: stop any one station and the belt still delivers — that robustness is the residual stream doing its job.