GEMM (General Matrix Multiply)

GEMM, short for General Matrix Multiply, is the core dense linear-algebra operation defined as C = αAB + βC, where A, B and C are matrices and α, β are scalars. It is the workhorse routine of the BLAS (Basic Linear Algebra Subprograms) standard, and it is the operation that consumes the overwhelming majority of arithmetic in modern AI workloads. Every fully-connected layer, attention projection, and — after an im2col transform — every convolution is ultimately cast as a GEMM.

Why it dominates AI compute

A single transformer layer multiplies large matrices many times during both training and inference, and a full model stacks dozens of such layers. Because GEMM is so central, hardware vendors design accelerators specifically to run it fast: GPU tensor cores and other systolic-array units are purpose-built matrix-multiply engines that do little else efficiently. When you see a chip rated at hundreds of TFLOPS, that headline number is almost always its peak GEMM throughput at a given numeric precision, not a general-purpose figure. In practice, if you can make the GEMMs fast, you have made the model fast, which is why so much low-level optimization effort is aimed squarely at this one routine.

Performance and tiling

Naive GEMM is bottlenecked by memory traffic, because it re-reads the same values from slow memory over and over. High-performance libraries split the work into cache-sized tiles and reuse each loaded value many times before discarding it. This data reuse is what lifts GEMM from a memory-bound to a compute-bound regime, letting it approach the hardware's peak floating-point rate. The amount of arithmetic done per byte fetched is the operation's arithmetic intensity, which determines where it sits on the roofline model of a given chip and therefore whether it is limited by math or by memory.

Precision and the shape of the matrices

GEMM performance depends heavily on numeric precision and on the shapes involved. Lower-precision formats such as FP16, BF16, or INT8 move fewer bytes and pack more operations into each tensor-core instruction, which is why quantization speeds inference. Shape matters too: a “tall-skinny” or single-token matrix — common during decode, when the batch is one — has little reuse and stays memory-bound no matter how fast the chip is. This is a large part of why generating one token at a time is so much less efficient than processing a whole prompt at once.

Why it matters for self-hosting

Understanding GEMM is the foundation for reasoning about real AI hardware efficiency. It explains why lower-precision formats speed things up, why a card's advertised TFLOPS rarely translates to that fraction of real-world speed once memory limits bite, and why batch size changes everything. For a sovereign operator choosing hardware, thinking in terms of GEMM throughput and memory bandwidth together predicts real performance far better than any single spec-sheet number.

A helpful way to internalize why GEMM rules AI is to notice that almost every trick for making models faster is, underneath, a trick for making GEMMs faster. Quantization shrinks the numbers a GEMM multiplies; batching enlarges the matrices so each has more reuse; specialized cores add hardware that does nothing but multiply and accumulate matrices; and fused kernels wrap a GEMM together with the operations around it so intermediate results never spill to slow memory. Even attention, the signature operation of transformers, decomposes into a pair of matrix multiplies with a softmax between them. Once you see a model as a long chain of GEMMs of varying shapes, its performance stops being mysterious: the fat, well-shaped multiplies fly near peak throughput, and the thin, single-token ones during generation are the ones that keep a machine memory-bound.

Understanding GEMM connects to the related concepts of FLOPS and arithmetic intensity, and in practice every GEMM on a GPU is dispatched as a tuned GPU kernel.