Gradient Checkpointing

Gradient checkpointing, also called activation checkpointing, is a training technique that trades extra computation for reduced memory. During an ordinary forward pass, a network stores every intermediate activation so they are available for the backward pass. For large models these activations — not the weights — often dominate memory use. Gradient checkpointing keeps only a strategic subset of activations and frees the rest, then recomputes the discarded ones on the fly during backpropagation, starting from the nearest saved checkpoint. Nothing about the mathematics changes; the gradients are identical. You simply refuse to remember what you can cheaply re-derive.

The compute-for-memory bargain

Because freed activations must be regenerated, the forward computation for checkpointed regions effectively runs twice. In transformer training the common pattern is to checkpoint at layer boundaries: store each layer's input, discard everything inside the layer, and replay the layer forward when its gradients are needed. In practice this can cut activation memory by several times at the cost of a training-throughput hit typically in the 25–35% range. Whether that trade is good depends entirely on what you are short of — and for a sovereign builder training on a single consumer GPU, the answer is almost always VRAM. A 30% slower run that fits is infinitely faster than a run that dies with an out-of-memory error at step one. Checkpointing is frequently the difference between "cannot fine-tune this model on my card" and "can, overnight."

Selective recomputation

The naive version recomputes everything inside a checkpointed span, but not all activations cost the same. Modern frameworks offer selective activation checkpointing, which preserves the outputs of expensive operations — large matrix multiplications, attention — while recomputing only cheap pointwise operations like activation functions and normalization. This recovers most of the memory savings at a fraction of the speed penalty, and the choice of what to keep is increasingly automated in PyTorch and similar stacks. The tuning question shifts from "checkpointing on or off" to "how aggressively," a dial you turn until the job fits with a little headroom.

Stacking it with everything else

Gradient checkpointing composes cleanly with the rest of the memory-reduction toolkit, and serious low-budget training uses them together. Sharded approaches like FSDP and ZeRO split weights, gradients, and optimizer state across devices but do little for per-GPU activation memory — checkpointing covers exactly that gap. Gradient accumulation lets you build a large effective batch out of micro-batches that fit; checkpointing makes each micro-batch cheaper still. Add mixed precision and parameter-efficient methods like LoRA, and the full stack is what makes single-GPU and small-rig fine-tuning of multi-billion-parameter models feasible at all.

The craftsman's framing

Checkpointing is an engineering trade in the classic sense: you have surplus compute and scarce memory, so you spend the surplus to cover the scarcity. It is the same reasoning a miner applies when underclocking a machine to fit a circuit's power budget — the resource you are constrained by dictates the tune, not the spec sheet's maximum. Know which resource binds you, and spend the other one.

Two practical notes for anyone flipping the switch. First, measure before and after: frameworks expose peak-memory statistics, and checkpointing's real-world savings depend on model architecture, sequence length, and batch size, so the only trustworthy numbers are from your own run. Longer sequences shift more memory into activations, which makes checkpointing disproportionately valuable for long-context fine-tuning. Second, remember it interacts with debugging: because activations are recomputed, any nondeterminism inside a checkpointed region can make gradients subtly irreproducible between runs, so pin your seeds and use deterministic kernels when chasing training bugs. Neither caveat changes the verdict — for memory-bound training on owned hardware, checkpointing is usually the first dial to turn and the last to regret.