Gradient Accumulation

Gradient accumulation is a training technique that lets you use a large effective batch size when your hardware cannot hold one in memory. Instead of running a single forward and backward pass over a huge batch, you process several smaller micro-batches in sequence, sum their gradients without touching the weights, and only perform one optimizer step after the last micro-batch. The model receives the same total gradient signal per update as a large batch would, but only one micro-batch ever occupies memory at a time. It is the simplest memory lever in deep learning: nearly free to implement, mathematically clean for most architectures, and the first thing to reach for when a fine-tuning run hits an out-of-memory error.

Effective batch size

The effective batch size equals the micro-batch size multiplied by the number of accumulation steps (and, in distributed settings, by the number of data-parallel workers). A micro-batch of 8 accumulated over 4 steps behaves like a batch of 32 for the purposes of the weight update while requiring memory for only 8 samples' activations. One practical detail hides in the arithmetic: each micro-batch's loss must be scaled by the number of accumulation steps (dividing by N) so the summed gradient matches what a true large batch would produce — modern training frameworks handle this automatically, but hand-rolled loops that skip it silently train with an inflated learning rate. Learning-rate schedules and warmup should be reasoned about in terms of the effective batch, not the micro-batch.

Costs and caveats

The trade-off is wall-clock time, not memory: you run multiple forward and backward passes per optimizer step, so each update takes proportionally longer, though total compute per epoch is essentially unchanged. The subtleties are few but real. Batch-normalization statistics are computed per micro-batch, not over the full effective batch, so architectures relying on BatchNorm see genuinely different normalization behavior — transformers, which use layer normalization, are immune to this and accumulate cleanly. And recent analyses note that accumulation is not always a perfectly free substitute for a genuinely larger batch at very small micro-batch sizes. Used with those caveats in mind, it remains the best memory-for-time trade available.

The sovereign fine-tuner's workhorse

Gradient accumulation is what makes fine-tuning on a single consumer GPU practical at all. A home machine with one card holds the model weights, optimizer state, and activations in a fixed VRAM budget; without accumulation, the batch sizes that fit would often be too small for stable training. With it, a micro-batch of 1 or 2 accumulated over 16 or 32 steps reproduces the training dynamics of hardware you did not have to rent. That is the sovereignty story in miniature: the technique moves a capability from the datacenter to the desk, the same direction every layer of a self-hosted stack should move. It stacks cleanly with the rest of the memory toolkit — combine it with gradient checkpointing to shrink activation memory, loss scaling and master weights for mixed-precision stability, and sharded training such as Fully Sharded Data Parallel (FSDP) when multiple devices are available. Each technique buys memory with a different currency — accumulation pays in time, checkpointing in recomputation, sharding in communication — and a well-tuned local training run usually spends all three.

A worked example makes the knobs concrete: suppose a recipe calls for an effective batch of 64 and your card fits a micro-batch of 4 — set accumulation to 16 and keep the recipe's learning rate. If you later fit a micro-batch of 8, halve the accumulation steps and change nothing else. Treating the effective batch as the invariant, and micro-batch times accumulation as an implementation detail, is the habit that keeps runs reproducible across different hardware — which matters the day you rent a bigger machine to scale up a run you prototyped at home.