HBM (High-Bandwidth Memory)

High-Bandwidth Memory (HBM) is a JEDEC-standard memory technology that stacks DRAM dies vertically and connects them to the processor over an extremely wide interface. Instead of a handful of memory chips spread around the board and wired through narrow channels, HBM piles dies into a 3D stack, threads them together with through-silicon vias (TSVs), and mounts the stack on the same package as the GPU or accelerator, linked by a silicon interposer. The result is far more memory bandwidth than conventional GDDR memory can deliver — exactly what data-hungry neural networks need.

Width over speed

Ordinary graphics memory chases bandwidth by clocking a narrow bus very fast, which costs power and signal integrity. HBM inverts the trade: a single stack exposes a 1024-bit interface — sixteen times wider than a GDDR channel — run at comparatively modest pin speeds, with the physical closeness of the interposer keeping signal paths short and energy per bit low. Bandwidth is width times speed, so the wide-and-slow approach wins on both throughput and power efficiency, at the cost of expensive advanced packaging. That packaging is also why you cannot upgrade it: HBM is bonded to the processor package at the factory, and the capacity you buy on day one is the capacity the card dies with.

The technology is not new so much as newly indispensable. AMD and SK hynix co-developed the first generation, which shipped on a consumer graphics card back in 2015; what changed is that transformer-era models made memory bandwidth the binding constraint, pulling HBM from an exotic option into the defining component of AI accelerators — and the single largest line item in their bill of materials.

Bandwidth is only half the stack's contribution; capacity rides along with it. Stacking DRAM dies eight and twelve high puts tens of gigabytes on a single stack, so a multi-stack accelerator carries model weights that would be flatly impossible to fit in conventional graphics memory — the reason frontier models are served from HBM parts even when raw compute would allow cheaper silicon.

Generations and bandwidth

The standard has advanced quickly. HBM3, published by JEDEC in January 2022, reaches data rates up to 6.4 Gbps per pin for about 819 GB/s per stack across the 1024-bit interface. HBM3E, the extended revision finalized in 2023, pushes pin speeds toward 9.8 Gbps and roughly 1.2 TB/s per stack. HBM4, standardized in 2025, doubles the interface to 2048 bits and lifts per-stack bandwidth above 2 TB/s. Flagship accelerators mount several stacks, so aggregate figures in the multiple terabytes per second are now routine on data-centre parts — each generation chipping away at the "memory wall" that otherwise leaves compute cores idling while they wait for data.

Why it matters — and what it means for self-hosters

In both training and inference, performance is frequently bounded not by arithmetic but by how fast weights and activations can be streamed into the compute units; large-model inference in particular is memory-bandwidth-bound, which is why HBM-equipped parts with their tensor cores command such premiums and why HBM supply — dominated by a small handful of memory fabs — has become a strategic chokepoint for the entire AI industry. That concentration should sound familiar to Bitcoiners: it rhymes with ASIC-foundry dependence in mining, another case of a critical layer pooling in very few hands.

For the on-premise AI builder, the practical reading is calibration, not despair. Consumer GPUs use GDDR, not HBM — but a used workstation card with wide GDDR and generous VRAM, feeding a well-quantized model (see quantization), delivers entirely respectable local inference. HBM explains the ceiling of the possible; it is not the price of admission to running your own models on your own hardware.