Multi-Query Attention (MQA)

Multi-Query Attention (MQA) is a modification of the standard multi-head attention used in transformer language models. In ordinary multi-head attention, every query head has its own matching key head and value head. MQA keeps the multiple query heads but collapses all of the key and value projections down to a single shared key head and a single shared value head. Proposed by Noam Shazeer in the 2019 paper "Fast Transformer Decoding," it was designed for one purpose: making autoregressive inference — the token-by-token generation loop — dramatically cheaper on real hardware.

The memory bottleneck it attacks

During generation, the slow part is not arithmetic but memory traffic. For every new token, the model must read the stored keys and values of all previous tokens — the KV cache — back from GPU memory, layer by layer. In a full multi-head model that cache holds one key/value pair per head per layer per token, and streaming it through the memory bus dominates decode latency. Because MQA stores only one key/value pair per layer instead of one per head, it shrinks the cache by a factor equal to the head count — often 32× or more on wide models. Less stored state means less memory bandwidth consumed per generated token, which translates directly into lower latency, longer contexts in the same VRAM, and more simultaneous requests per card.

What the collapse costs

Forcing every query head to look through the same key/value projection removes representational capacity: heads can no longer specialize their notion of what to attend to, only how to query it. Models trained with pure MQA show measurable quality degradation on some tasks and a documented tendency toward training instability compared with full multi-head attention. The trade was often worth it for serving economics, and several production models — notably early PaLM and Falcon variants — shipped with MQA. But the quality gap is precisely why Grouped-Query Attention exists: GQA keeps a small number of shared key/value groups instead of exactly one, recovering nearly all the quality while retaining most of the memory savings, and it has since displaced pure MQA as the default in open-weight models.

Why a self-hoster should care

For an operator running open-weight models on their own hardware — the same sovereignty instinct that puts a Bitcoin node in the basement instead of trusting a hosted API — the attention variant a model uses is not academic trivia. It determines the KV-cache size per token, which sets how much context fits in your GPU before generation slows or fails. A pure-MQA model can hold an order of magnitude more context in the same VRAM than an equivalent full multi-head model, and it will decode faster at long context because each step moves fewer bytes. Reading a model card's head configuration (query heads versus key/value heads) tells you which regime you are in: equal counts mean full attention, a KV count of one means MQA, anything between means GQA.

Position on the efficiency spectrum

MQA also changes the economics of batching. Because the single key/value projection is shared across heads, serving frameworks can pack more concurrent sequences into a fixed cache budget, and throughput-oriented deployments feel the benefit even more than single-user latency does. The savings compound with cache quantization — storing keys and values at reduced precision — since both techniques attack the same quantity: total bytes of cached state that must survive in fast memory for the life of each session.

MQA is best understood as one endpoint of a dial. Full multi-head attention sits at maximum quality and maximum memory cost; MQA sits at minimum memory cost with a real quality tax; GQA occupies the tunable middle and is where the ecosystem settled. All three variants benefit further from cache-management techniques and quantization, and all are constrained by the same underlying reality — the memory wall — that makes moving bytes, not multiplying numbers, the scarce resource in local inference.