Arithmetic Intensity

Arithmetic intensity is the ratio of arithmetic work to data movement in a computation, usually expressed as floating-point operations per byte transferred (FLOPs/byte). It is the single most useful number for predicting how a workload will perform on a given accelerator, because it tells you in advance which of the chip's two fundamental limits — the compute units or the memory system — the workload will slam into first. Spec sheets advertise TFLOPS; arithmetic intensity tells you whether you will ever see them.

Why the ratio decides everything

Every operation needs operands fetched from memory and results written back. If a kernel performs many calculations per byte moved — high arithmetic intensity — the compute units stay saturated and performance is limited by peak FLOPS. If it performs little arithmetic per byte, the chip idles waiting on memory, and performance is capped by memory bandwidth no matter how many TFLOPS the datasheet claims. Every processor has a break-even intensity — its peak compute divided by its peak bandwidth — and on modern accelerators that number is punishingly high, because compute throughput has grown far faster than memory systems for decades. Most real workloads sit below the line, which is why "memory-bound" is the default condition of modern computing, not the exception.

Examples in AI

A large matrix multiplication with good tiling reuses each loaded value across an entire row or tile of the output, giving high arithmetic intensity — this is why training and batched workloads utilize accelerators well. By contrast, LLM inference for a single user is dominated by the decode phase: generating each token requires streaming essentially the entire weight set through the chip while performing only a few operations per weight. That is very low arithmetic intensity, which yields the practical rule of thumb for local LLMs: single-stream tokens per second is approximately memory bandwidth divided by model size in memory. This is precisely why a sovereign builder choosing a local-inference machine should weigh memory bandwidth above almost everything else, and why quantization speeds up inference — shrinking the bytes per weight raises the effective intensity of the same math. The prefill phase, which processes the whole prompt at once, has far higher intensity, which is why prompt ingestion is fast while generation plods; batching multiple requests raises decode intensity too, since one pass of streamed weights serves many tokens — the lever behind continuous batching on serving hardware.

A mining-flavored footnote

Bitcoin mining sits at the opposite extreme: SHA-256 hashing needs almost no memory traffic at all — a tiny block of state, hashed over and over. Its effectively unbounded arithmetic intensity is exactly why the work moved to ASICs: with no memory bottleneck, pure custom logic wins outright, and joules per hash becomes the only scoreboard. Comparing that with a memory-starved LLM decode is a compact education in why hardware must fit the workload.

Arithmetic intensity is the x-axis of the roofline chart. See the roofline model and the closely related framing of operational intensity to put the number to work.

Intensity can also be engineered, not just measured. Kernel fusion — performing several operations on data while it is already in registers — raises FLOPs per byte without changing the math, and the best-known attention optimizations restructured their algorithms around exactly this idea. On the buying side, the metric gives a one-question test for any hardware pitch: what is my workload's intensity, and which side of this chip's ridge point does it land on? Vendors quote peak numbers; the ratio tells you which peak you will actually touch. It is the rare piece of performance theory that pays for itself the first time it stops a mispurchase.