Memory Bandwidth

Memory bandwidth is the rate at which a processor can move data between its compute cores and its memory, measured in gigabytes per second (GB/s). A useful mental model: if VRAM is a warehouse whose size is its capacity in gigabytes, memory bandwidth is the width of the road leading to it. A wide road keeps the compute cores fed; a narrow one leaves them idle, no matter how many teraflops the spec sheet brags about.

The memory wall

Modern AI hardware is frequently limited not by arithmetic but by data movement. Compute performance has grown much faster than memory speed for decades, so processors increasingly sit starved: in large-scale training, typical model-FLOPs utilization lands around 30–50% because GPUs stall waiting on memory and interconnect. In inference the effect is even sharper. Generating each token of an LLM response requires streaming essentially all of the model's weights (plus the growing attention cache) out of memory and past the cores — and that traffic, not the math, sets the pace. This is the "memory wall": once you are bandwidth-bound, adding more FLOPS does nothing.

The napkin math every self-hoster should know

Because token generation is a weight-streaming exercise, you can estimate speed with one division: tokens per second ≈ memory bandwidth ÷ model size in bytes. A machine with 400 GB/s of bandwidth running a model that occupies 20 GB can generate at most about 20 tokens per second, full stop — and real systems land somewhat below the ceiling. This single equation explains most of what confuses newcomers to local AI: why quantization speeds up generation (fewer bytes per weight means fewer bytes to stream per token), why a GGUF model at 4 bits runs roughly twice as fast as at 8 bits on the same card, why Apple-silicon Macs with fast unified memory punch far above their GPU-compute weight for local-LLM work, and why splitting a model onto CPU-attached RAM — often several times slower per byte than GPU memory — craters throughput. Prompt processing, by contrast, batches many tokens through the weights at once and leans back toward compute, which is why long-context ingestion and generation can feel like different machines.

How the hardware delivers it

Bandwidth is the product of bus width and memory speed, and the market segments cleanly along it. Ordinary desktop DDR5 delivers on the order of tens of GB/s; consumer GPUs with GDDR memory reach several hundred to about a terabyte per second; and datacenter accelerators pair their cores with HBM — stacked memory on an extremely wide bus — to reach multiple terabytes per second. You largely get what you pay for, and what you are paying for is the road, not the warehouse.

Buying advice, craftsman style

The single-user napkin math does bend under load. Serving many requests at once lets the runtime batch them, streaming the weights once per step for the whole batch — so aggregate throughput climbs well past the single-stream ceiling even as each individual session still feels bandwidth-bound. This is why datacenter serving economics differ from homelab feel, and why a family or small-team server benefits from batching-aware runtimes. For the common case that matters here — one operator, one conversation, local hardware — the simple division remains the honest predictor.

When sizing hardware for an on-premise AI rig you intend to use daily, read the memory-bandwidth line before the TFLOPS line: for interactive LLM use it predicts real-world token speed far better. Capacity decides which models you can load; bandwidth decides how they feel. A miner would recognize the discipline — nobody buys an ASIC on chip count alone; you buy the whole machine's real throughput per dollar and per watt.