Expert Parallelism

Expert parallelism is a parallelization strategy built specifically for Mixture-of-Experts (MoE) architectures. An MoE layer contains many small sub-networks called experts, but each token is routed to only a few of them, so most of the layer's parameters sit idle for any given token. Expert parallelism distributes those experts across multiple accelerators, so no single device has to hold them all, and dynamically sends each token to whichever device hosts the expert it was assigned.

Routing and all-to-all

The defining communication pattern is the all-to-all collective. Before the MoE layer runs, a gating network decides which expert each token belongs to, and an all-to-all dispatch shuffles tokens across devices so they land where their experts live. After the experts compute, a second all-to-all gathers the results back to their original positions. This dispatch-and-combine pair is the performance bottleneck of expert parallelism, because it moves data across the interconnect twice per layer, and the whole layer can only proceed as fast as that shuffle completes. Specialized token dispatchers are used to overlap that communication with computation and to scale routing efficiently to dozens of expert-parallel devices, hiding as much of the network latency behind useful work as possible.

Load balancing

Because routing is data-dependent, some experts can receive far more tokens than others, leaving devices unevenly loaded and forcing the whole layer to wait on the busiest one. MoE training therefore relies on auxiliary load-balancing objectives and capacity limits that cap how many tokens any one expert will accept, dropping or rerouting the overflow. Get the balance wrong and the system either wastes accelerators sitting idle or silently discards tokens, quietly hurting quality; get it right and the sparse model runs close to the efficiency its parameter count promises. This balancing act is one reason MoE models are trickier to train well than their dense counterparts.

Layering with other parallelism

Expert parallelism is rarely used alone. Large sparse models combine it with data, tensor, and pipeline parallelism so that experts, layers, tensors, and batches are all split in complementary ways across a cluster. The art is choosing how many devices to devote to each axis: too much expert parallelism and the all-to-all traffic dominates; too little and the experts do not fit in memory. Serving frameworks increasingly expose these axes as tunable knobs, because the right split depends heavily on the specific model, batch size, and interconnect.

Why it matters for self-hosters

For a sovereign operator, the practical takeaway is that MoE models trade memory for communication — they fit more total parameters per accelerator but demand a fast interconnect to move tokens between experts, which shapes whether a given rig can serve them well. A single consumer GPU can run a small MoE, but scaling one across several cards rewards a good link between them far more than a dense model of the same size would.

Communication cost is the theme that ties expert parallelism together. The all-to-all shuffles it depends on are sensitive to the physical topology of the machine: cards linked by a fast, dedicated interconnect can exchange tokens far more cheaply than cards forced to talk over a slower general-purpose bus. This is why the technique is most at home in tightly-coupled clusters, and why naively spreading experts across loosely-connected consumer machines often performs worse than simply running a smaller dense model. Recent serving engines invest heavily in overlapping the dispatch and combine steps with computation, and in kernels that fuse routing with the expert math, precisely because that shuffle is the ceiling on how well a sparse model scales. Understanding that ceiling helps a builder decide whether an MoE model is a good fit for the hardware they actually own.

Expert parallelism sits alongside the core axes of Data Parallelism, Tensor Parallelism, and Pipeline Parallelism in a full distributed stack, and it exists to scale the same sparse routing that Mixture of Tokens was proposed to stabilize.