Tensor Parallelism

Tensor parallelism, also known as horizontal or intra-layer parallelism, divides the math inside a single neural-network layer across multiple accelerators. Instead of giving each device a copy of the whole model, the large weight matrices of a layer are sliced column-wise or row-wise, each device computes its partial result against its slice, and the pieces are combined into the full output. This is the strategy that lets a single layer too large for any one accelerator's memory run at all — and it is equally a latency tool, since several devices attack the same matrix multiplication simultaneously.

How the work is split

In a transformer, the attention and feed-forward matrix multiplications are the natural targets. The scheme popularized by Megatron-style training splits paired layers in complementary directions — the first weight matrix column-wise, the second row-wise — so that an entire feed-forward block or attention block needs only one synchronization at its boundary rather than after every multiply. Attention heads shard especially cleanly, since each head is already an independent computation. Even so, an all-reduce is required within every transformer block, in both the forward and backward pass. Because this synchronization sits on the critical path of every single layer, tensor parallelism carries the highest communication overhead of the common strategies and is most effective inside a single node where devices share a high-bandwidth interconnect such as NVLink; stretch it across ordinary networking and the accelerators spend their time waiting instead of computing.

Where it fits among the strategies

Tensor parallelism rarely stands alone at scale. It is typically combined with pipeline parallelism (splitting the model by layers into stages) and data parallelism (replicating the whole stack and splitting the batch) into what practitioners call 3D parallelism — the layout used to train the largest models. The rule of thumb: keep tensor parallelism inside a node where bandwidth is abundant, use pipeline parallelism across nodes to conserve interconnect, and lay data parallelism over the top for throughput. Sharded-optimizer approaches like FSDP and ZeRO attack the same memory problem from a different angle — partitioning storage rather than computation — and are often the simpler choice when the goal is fitting a model rather than accelerating a single layer.

Why it matters for local inference

Tensor parallelism is not just a training concept. Local inference stacks use it to pool the memory of multiple consumer GPUs: a model too large for one card's VRAM can run split across two or four, each holding a slice of every layer. The catch is the same synchronization tax — over PCIe instead of NVLink, per-token latency takes a real hit, and a multi-GPU tensor-parallel rig rarely scales as well as its combined spec sheet implies. For a sovereign builder, the honest guidance is: prefer one GPU with enough memory, use quantization to make the model fit before reaching for a second card, and treat tensor parallelism as the tool for when a model genuinely cannot fit any other way. It is the same lesson mining teaches about memory bandwidth and interconnects: the spec-sheet compute means nothing if the data cannot reach it.

Compare pipeline parallelism, which splits the model by layer, and data parallelism, which replicates the model and splits the data.

One more asymmetry is worth knowing before buying hardware: tensor parallelism splits the weights evenly, but it also partitions the KV cache and duplicates some activations, so the usable memory gained per added GPU is less than naive division suggests — and on consumer PCIe rigs, two-card setups commonly deliver well under twice the single-card throughput. Serving frameworks document their effective scaling; reading those numbers before buying a second card is cheaper than discovering them after. As with every distributed scheme, measure the actual tokens per second, not the theoretical FLOPS.