Data Parallelism

Data parallelism is the most common way to train a neural network across multiple accelerators. Every device holds a complete copy of the model, and each training batch is split into shards so that every device processes a different slice of the data simultaneously. After the backward pass, the per-device gradients are averaged across all devices — an all-reduce operation — so every replica applies the same update and stays numerically identical, stepping in lockstep. Conceptually it is the simplest distribution strategy: same model everywhere, different data everywhere, synchronize once per step.

Why it scales so well

Because each device runs a fully independent forward and backward pass and only needs to communicate gradients once per step, data parallelism has modest communication overhead relative to strategies that split the model itself. The all-reduce can also be overlapped with computation — gradients for early layers are exchanged while later layers are still back-propagating — which hides much of the communication cost on decent interconnects. It is the default in mainstream frameworks (PyTorch's DistributedDataParallel, for example), and throughput grows close to linearly with device count for models that fit comfortably in memory. That combination of simplicity and near-linear scaling is why data parallelism remains the backbone of most training runs, from two consumer GPUs in a homelab to thousand-accelerator clusters.

The memory ceiling

The catch is memory. Every device must hold the entire model — weights, gradients, and optimizer state, which for common optimizers can multiply the weight footprint several times over — plus activations for its batch shard. Adding devices adds throughput but does not add per-model memory: ten GPUs data-parallel can train a model no larger than one GPU can hold. Once a model outgrows a single accelerator's VRAM, plain data parallelism simply stops working. That wall is exactly what the sharded variants were built to break: Fully Sharded Data Parallel (FSDP) and the ZeRO family partition weights, gradients, and optimizer states across devices, gathering shards only when needed, trading extra communication for a memory footprint that shrinks with device count. Model-splitting strategies — tensor parallelism (splitting individual layers across devices) and pipeline parallelism (splitting the layer stack into stages) — sidestep the ceiling differently by never materializing the whole model anywhere. Large training runs routinely combine all of these, but data parallelism is almost always the outermost layer.

Batch size, and a homelab trick

Data parallelism multiplies the effective batch size by the device count, which usually helps but interacts with learning-rate tuning — a detail worth knowing before assuming more GPUs equals strictly better training. The inverse trick matters more on owned hardware: gradient accumulation runs several forward/backward passes and sums gradients before stepping, simulating a large batch on a single device. It is data parallelism unrolled in time instead of across hardware — slower, but free.

For the sovereign builder

If you are fine-tuning open models on hardware you own, data parallelism is the first lever to pull when you outgrow one GPU: it requires no model surgery, works over ordinary networking for modest scales, and fails loudly rather than subtly. The decision tree is short. Model fits on one card? Data parallel across however many you have. Model does not fit? Reach for FSDP/ZeRO sharding or quantized training before renting cloud time. Only at scales well beyond a homelab do tensor and pipeline parallelism earn their complexity. Owning the training loop end to end — data, model, and the hardware it runs on — is the machine-learning expression of the same instinct that puts a miner in the garage instead of a hosting contract.