Throughput-Optimized Serving

Throughput-optimized serving is a way of configuring a large language model inference server to maximize the total number of tokens produced per second across all concurrent requests, rather than minimizing the response time of any single one. It sits at one end of the fundamental throughput-versus-latency trade-off that governs all LLM serving: the same GPU can serve one user very fast or many users somewhat slower, and the operator must choose where on that curve to run.

Why the trade-off exists

Text generation has two phases with opposite hardware characteristics. Prefill processes the entire prompt in one pass and saturates GPU compute — it is arithmetic-bound and parallel. Decode then emits one token at a time, and each step must re-read the model's weights from memory to produce a single token per sequence; it is memory-bandwidth-bound, leaving the GPU's compute units mostly idle unless many requests share the step. Batching is the fix: when dozens of sequences decode together, the expensive weight loads are amortized across all of them, and total tokens per second climbs steeply. But every request in the batch now waits for the batch's step time, so each individual user sees tokens arrive more slowly. Bigger batches mean higher aggregate throughput and higher per-user latency; smaller batches mean the reverse. There is no configuration that maximizes both.

Goodput, not raw throughput

A server tuned purely for token volume can technically hit huge numbers while every user's chat feels broken. The metric that matters in practice is goodput: the highest request rate the server sustains while still meeting its service-level objectives on time-to-first-token and time-per-output-token. Modern serving stacks offer a toolbox for pushing throughput up without blowing past those targets: in-flight batching admits and retires requests continuously instead of waiting for a full batch to drain; chunked prefill slices long prompts into pieces interleaved with decode steps so one giant prompt cannot stall everyone else's token stream; and request scheduling policies decide who gets admitted when memory for the KV cache runs tight. Disaggregated designs go further and run prefill and decode on separate hardware pools sized independently.

Tuning it on your own hardware

For a sovereign operator with one or a few GPUs, the same trade-off applies at small scale, and the constraint that binds first is usually VRAM: batch size is capped by how many sequences' KV caches fit alongside the weights, which is why quantization and context-length discipline directly buy you throughput. A throughput-leaning profile — large maximum batch, aggressive admission — is the right choice for batch workloads: overnight document processing, bulk embedding and classification jobs, dataset generation, summarizing a year of logs. A latency-leaning profile — small batch, fast admission — is right for interactive chat, where a human is watching the tokens arrive. Many self-hosters run both: an interactive endpoint tuned for snappy first tokens, and a batch queue that soaks up idle GPU time at maximum efficiency.

The economics

Throughput optimization is ultimately about cost per token, and it is where self-hosted inference earns its keep. A GPU that sits at 15 percent utilization serving one request at a time is wasting most of the capital you paid for it; the same card running well-batched workloads can produce several times the tokens per watt. Miners will recognize the shape of this reasoning instantly — it is the same discipline as chasing joules per terahash: the hardware is a fixed cost, the energy is the ongoing cost, and the operator's job is to extract the maximum useful output from both. The knob is different; the mindset is identical: measure, adjust, re-measure — serving configurations reward the same patient empiricism as miner tuning.