Activation Recomputation

Activation recomputation, also called gradient or activation checkpointing, is a memory-saving technique that trades extra computation for reduced memory during neural-network training. Backpropagation normally needs the intermediate activations from every layer's forward pass in order to compute gradients, and for deep models or long sequences those stored activations can consume more memory than the model's own weights. Recomputation lets you throw most of them away and rebuild them only at the moment they are actually needed.

Save now, recompute later

Instead of keeping every activation, the technique stores only a sparse subset — the checkpoints — and discards the rest. During the backward pass, whenever a missing activation is required, it is regenerated on the fly by re-running the forward computation from the nearest stored checkpoint. For a network of L layers, checkpointing roughly every √L layers cuts activation memory from order-L down to order-√L, a reduction that pays off more the deeper the model gets. The idea is old and well understood, which is why it appears under several names; you will see the exact same mechanism described as gradient checkpointing in most training frameworks, where it is usually a single flag you toggle per layer or per block. Crucially, it is mathematically transparent: the recomputed activations are identical to the originals, so the final trained weights are unchanged.

The compute trade-off

The price is that the forward computation for non-checkpointed layers runs twice — once during the original forward pass and again during backward regeneration. In practice this typically adds on the order of 20% to 30% to training time. In exchange, the freed memory lets you fit a larger model, or — often more valuable — a larger batch size on the same GPU, which can improve throughput and gradient stability elsewhere in the run. It is a lever you reach for when memory, not raw compute, is the binding constraint, and transformer training with long context windows is exactly the regime where activations dominate the memory budget.

Choosing what to keep

Not every layer is equally worth recomputing. Cheap operations that produce large activations — normalization, dropout, activation functions — are ideal to discard and regenerate, because re-running them costs almost nothing yet frees a lot of memory, while expensive matrix multiplies are often better kept. Modern frameworks expose this as selective or fine-grained checkpointing, letting a practitioner tune the memory-versus-time curve per block rather than accept a single blunt setting. Placement therefore becomes a real optimization: too few checkpoints and the backward pass recomputes enormous spans; too many and you have simply reintroduced the memory cost you were trying to avoid. The √L spacing is a useful default, but the best configuration depends on the model's shape and on exactly where the memory pressure actually sits. It is also worth checking that recomputation is genuinely helping: a quick profile of the run reveals whether memory or compute is the true bottleneck, and there is little point paying the recompute tax on a job that was never memory-bound in the first place.

Why it matters on self-owned hardware

Recomputation is one of the few techniques that lets a modest, self-hosted rig punch above its weight, so a builder training or fine-tuning a model on a single consumer GPU can attempt runs that would otherwise demand data-center memory. It stacks cleanly with other memory savers rather than competing with them: it is frequently combined with ZeRO-Offload / CPU offload, with reduced-precision formats like BF16, and with tensor parallelism once a model is spread across several cards. For the sovereign AI practitioner who refuses to rent someone else's cluster, stacking these tricks is what makes ambitious training feasible on hardware you actually own and control — keeping both the model and the data on your own bench instead of in a rented account you do not govern.