Compute-Bound vs Memory-Bound

Every workload running on a processor or accelerator is, at any given moment, limited by one of two resources: the speed of its arithmetic units or the speed of its memory system. A workload is compute-bound when performance is limited by how fast the chip can do math — the memory system keeps the data flowing, and the arithmetic units are the queue. It is memory-bound when the arithmetic units sit starved, waiting for operands to arrive from memory. Knowing which regime you are in is the first diagnostic act of performance engineering, because it tells you exactly what to optimize — and, just as important, which expensive upgrade would accomplish nothing.

The deciding factor

The dividing line is quantitative. Compute a kernel's arithmetic intensity — operations performed per byte of memory traffic — and compare it against the hardware's ridge point, its peak operations per second divided by its peak memory bandwidth. If the kernel's intensity exceeds that ratio, the chip can keep its math units fed and the workload is compute-bound; below it, the math units starve and the workload is memory-bound. This is the formal logic of the roofline model, and its measured refinement — counting only true off-chip traffic — is operational intensity. The comparison is machine-relative: the same kernel can be memory-bound on a chip with towering compute and modest bandwidth, yet compute-bound on a more balanced part. Neither regime is inherently bad; what is bad is misdiagnosing which one you are in.

What each regime prescribes

The regimes have different medicine cabinets. Compute-bound work responds to more or faster math: higher clocks, more parallel units, lower-precision number formats that multiply arithmetic throughput, better instruction mixes. Memory-bound work responds to moving fewer bytes: cache-friendly tiling, kernel fusion that keeps intermediates on-chip, data compression and quantization, and — on the hardware side — bandwidth, not TFLOPS. Apply compute-regime medicine to a memory-bound problem and you get nothing: the shiny new accelerator with double the TFLOPS but the same memory bandwidth delivers the same tokens per second, a lesson many home-lab builders have paid retail to learn. The wall between the regimes has a name and a history — see the memory wall — and decades of hardware evolution are best read as an arms race against it.

AI and mining through the same lens

The distinction illuminates both halves of this site's world. Training large models on big batches is typically compute-bound: enormous matrix multiplications reuse each fetched weight many times, so datacenter training parts chase peak FLOPS. Single-stream inference — the sovereign, self-hosted case — is firmly memory-bound: generating each token streams the whole weight set through the chip roughly once, so local performance tracks memory bandwidth almost linearly, and the right purchasing question for a local-AI box is "how fast is the memory?" rather than "how many TFLOPS?" Bitcoin mining, meanwhile, is the purest compute-bound workload in commercial existence: SHA-256 hashing touches essentially no memory, which is why mining ASICs are wall-to-wall arithmetic with no expensive memory subsystem at all — and why no amount of memory bandwidth ever helped anyone find a block. Same silicon industry, opposite ends of the intensity axis; diagnose before you spend.

The regime lens also predicts the future usefully: every architectural trend of the past decade — high-bandwidth memory, giant on-chip caches, low-precision formats, processing-in-memory research — is a direct response to more workloads sliding into the memory-bound regime. When you evaluate the next accelerator generation, skip the headline TFLOPS and ask what moved: if bandwidth and cache grew faster than compute, the designers are telling you where the real bottleneck lives. Reading hardware roadmaps through this one distinction is the cheapest analyst skill a self-hosting builder can acquire.