Grouped-Query Attention (GQA)

Grouped-Query Attention (GQA) is an attention mechanism that interpolates between two extremes: full multi-head attention, where every query head owns a private key and value head, and Multi-Query Attention, where all query heads share exactly one. GQA partitions the query heads into a small number of groups, and every head within a group shares a single key head and value head. Introduced by Google researchers in a 2023 paper, it rapidly became the default design in modern open-weight models — the Llama 2 70B and Llama 3 families, Mistral, Qwen, and many others all ship with GQA.

Two extremes, one dial

The elegance of GQA is that both older designs are special cases of it. Set the number of key/value groups equal to the number of query heads and every head gets a private pair: that is standard multi-head attention. Set the group count to one and every head shares a single pair: that is MQA. Everything between is a tunable trade-off. A typical configuration uses 8 key/value groups for 32 query heads — a 4× reduction in stored key/value state — and empirically lands within a whisker of full attention on quality benchmarks while capturing most of MQA's speed. The research showed the quality curve is steep near the MQA end and flat near the multi-head end, so a modest group count buys nearly everything.

Why it dominates open-weight models

The stored key/value state — the KV cache — is what a transformer must stream out of GPU memory for every generated token, and at long context it dwarfs the model weights as the thing limiting throughput. Cutting it 4–8× directly lowers per-token latency, lets more concurrent requests share one card, and stretches how much context fits in fixed VRAM. Just as important commercially: an existing multi-head checkpoint can be "uptrained" into a GQA model by mean-pooling its key/value heads and continuing training for a small fraction — on the order of 5% — of the original compute. The technique was therefore cheap to retrofit, which is exactly why it swept through the ecosystem so quickly.

What it means on your own hardware

For the self-hoster — the operator who runs models locally for the same reason they run their own Bitcoin node — GQA is usually the unglamorous reason a 70B-class model serves long contexts on prosumer GPUs at all. When you read a model card, compare num_attention_heads with num_key_value_heads: the ratio between them is the GQA group factor, and it tells you the KV-cache footprint per token before you download a single weight file. A model with 64 query heads and 8 KV heads stores one-eighth the cache of its full-attention equivalent, which can be the difference between a 32K context fitting in memory or spilling. That knowledge turns capacity planning from guesswork into arithmetic — bytes per token times context length times layer count.

Position in the design space

GQA also interacts kindly with the rest of the serving stack. Tensor-parallel deployments can shard one key/value group per device slice, cache offloading moves proportionally less data when a session is swapped out, and speculative decoding re-reads a smaller cache on every verification pass. None of these systems tricks required GQA, but all of them get cheaper under it — a reminder that a single architectural decision made at training time echoes through every layer of inference infrastructure afterward.

GQA reduces how much key/value state exists; it does not change the quadratic shape of attention itself. It therefore composes cleanly with the other levers of efficient inference — quantized caches, paged cache management, and the sparse or windowed attention patterns used in long-context architectures — and every one of those levers is ultimately about the same constraint: memory bandwidth is scarcer than compute. On the spectrum of attention designs, GQA is the settled middle ground between full multi-head attention and MQA, and for now it is the pragmatic answer most of the open-weight world has converged on.