Pipeline Parallelism

Pipeline parallelism is a model-partitioning strategy that splits a neural network by depth: its layers are divided into consecutive stages, and each stage is placed on a different accelerator. A batch flows through the stages like work moving down an assembly line — device one computes the first block of layers, hands its activations to device two, and so on to the output, with gradients flowing back along the same path in reverse during training. It is the natural answer when a model's weights are simply too large for any single GPU: instead of copying the model everywhere, you slice it lengthwise and let each device own one contiguous piece.

The bubble problem

Run naively, a pipeline wastes most of its hardware. While stage one processes the batch, every later stage sits idle; while the last stage works, the earlier ones wait. This idle time is the pipeline bubble, and shrinking it is the entire craft of pipeline scheduling. The standard fix is to split each batch into many micro-batches and feed them in staggered, so at any moment different stages are busy with different micro-batches — the same mechanics that make gradient accumulation work, repurposed to keep an assembly line full. The GPipe schedule runs all forward passes then all backwards; the 1F1B (one-forward-one-backward) schedule interleaves them, which sharply reduces how many activations must be held in memory at once; interleaved schedules go further by giving each device several smaller, non-contiguous stages. The arithmetic is unforgiving: the bubble fraction shrinks roughly with the ratio of micro-batches to stages, so deep pipelines demand many micro-batches to stay efficient. A second, quieter challenge is load balance — stages must take nearly equal time, because the slowest stage sets the pace for the whole line.

Communication profile and where it fits

Pipeline parallelism's great virtue is cheap communication. Each stage exchanges only its boundary activations with its immediate neighbors — point-to-point transfers, modest in size — rather than the massive, frequent all-reduce collectives that tensor parallelism requires inside every layer. That makes pipelining the right tool for spanning slower links: in the classic 3D-parallel recipe used to train large models, tensor parallelism runs inside a node where NVLink-class bandwidth is available, pipeline parallelism spans across nodes over the network, and data parallelism replicates the whole arrangement to scale batch throughput. Sharding approaches like FSDP attack the same memory problem from a different angle and are often simpler when the model merely almost fits.

Why home-lab builders should care

For the sovereign AI practitioner, pipeline thinking matters most at inference time. Splitting an open-weight model by layers across two or three consumer GPUs — or across a GPU and CPU memory — is exactly pipeline partitioning, and popular local-inference stacks do it as a matter of course. The bubble economics explain a common disappointment: with a single conversation (batch size one), a layer-split model uses only one device at a time, so two GPUs give you a bigger model, not a faster one. Throughput returns when you serve many requests concurrently — micro-batches by another name. Understanding that trade lets you buy hardware for the workload you actually run rather than the benchmark you read about.

The deeper appeal for sovereign builders is that pipelining is how modest hardware punches above its weight: it converts "this model does not fit" into "this model runs, with known trade-offs," using nothing but partitioning discipline. The same idea scales from a two-GPU desktop to the thousand-node clusters that trained the frontier models — which is a useful reminder that the big labs' techniques are not magic, just engineering, and most of it is published. Read the schedules, understand the bubble, and the hardware you already own gets measurably more capable.