Continuous Batching

Continuous batching is a scheduling technique for serving large language models that dramatically improves how many requests a GPU can handle at once. Traditional static batching groups requests together but waits for every request in the batch to finish before admitting the next group — wasteful, because responses finish at wildly different lengths and the GPU idles on the stragglers' behalf. Continuous batching instead makes scheduling decisions at every generation step — hence its other name, iteration-level scheduling — admitting new requests into the active batch the moment a slot frees up and retiring completed ones immediately. The GPU almost never sits idle.

Why throughput jumps

Autoregressive generation is a poor match for how GPUs want to work. Producing one token for one request uses a sliver of the chip's parallel capacity, because the step is bound by streaming weights through memory rather than by arithmetic. Batching many requests into each step amortizes that memory traffic across all of them: the weights are read once per step regardless of whether one request or fifty ride along. Static batching captures some of this but bleeds it back at the ragged end of every batch; continuous batching keeps occupancy high through the whole serving day. Published benchmarks for engines built around the technique report throughput gains on the order of ten to twenty times over naive request-at-a-time serving, with better tail latency under bursty load as a bonus — new arrivals no longer wait for an entire old batch to drain.

The memory-management partner

Packing many in-flight requests onto one GPU raises a second problem: each request carries a growing key-value cache, and naive allocators fragment VRAM badly enough to cap the batch size well below what the hardware could hold. Modern serving engines pair continuous batching with paged KV-cache management, which allocates cache memory in small blocks on demand — much like an operating system pages RAM. The two techniques together are what let a single card serve a surprising number of simultaneous streams; either one alone leaves most of the gain on the table.

Relevance to self-hosting

If your local model serves exactly one user — you, at a terminal — continuous batching barely matters, and a simple runtime is the right tool. It starts to matter the moment concurrency appears: a household sharing one AI box, a small team behind a chat interface, or several agents and pipelines hitting the same endpoint. That is the point where a single GPU either keeps up gracefully or falls over, and continuous batching is usually the difference — capacity you already paid for, unlocked by scheduling rather than by buying more hardware. It is a core feature of the popular open-source inference servers and generally engages automatically once you use one; the practical decision is simply choosing a serving engine rather than a single-user runtime when your workload becomes shared.

Part of a larger toolbox

Continuous batching composes cleanly with the rest of the serving stack: flash attention speeds up the attention computation itself, speculative decoding reduces the number of expensive steps per token, and quantization shrinks the weights everything else must stream. All of them serve the same sovereign goal: faster, denser local inference on hardware you own.

Measuring the benefit is straightforward if you separate the two numbers that matter: total throughput (tokens per second across all streams) and per-user latency (time to first token, then tokens per second within one stream). Continuous batching trades a little of the second for a lot of the first — each individual stream may run slightly slower than it would alone, while the machine's total output multiplies. For a shared box that trade is almost always right, and watching aggregate GPU utilization climb from a sliver to near-saturation under the same hardware is the clearest demonstration in all of local AI that scheduling, not silicon, was the bottleneck.