Sparse Attention

Sparse attention is a family of techniques that approximate full self-attention by letting each token attend to only a chosen subset of other tokens instead of the entire sequence. Standard attention is dense — every token attends to every other token — which costs O(n²) in sequence length and becomes prohibitive for long documents: doubling the context quadruples the work. Sparse attention replaces that dense pattern with a structured one, cutting cost toward linear while aiming to preserve most of the model's expressive power. It is one of the core tricks that made long-context transformers practical outside the datacenter.

Common sparse patterns

Practical designs mix a few building blocks. Local (window) attention connects each token to its near neighbours — cheap, and a good fit for language, where most dependencies are short-range. Global attention lets a handful of special tokens attend to, and be attended by, the whole sequence, giving the model a shared scratchpad for summary or task tokens. Random attention sprinkles in a few arbitrary long-range connections so information can hop between distant regions in a small number of steps. Longformer combined local and global attention to reach linear scaling on long documents; BigBird added the random links and showed theoretically that such sparse patterns can match the expressive power of full attention. Sliding window attention, used in several popular open-weight models, is the local pattern in its purest form, with deeper layers extending effective reach beyond the window.

What it costs

Sparsity is an approximation, and the failure mode is predictable: if a genuine dependency falls outside the pattern — a fact on page 2 needed on page 200, with no global token or random link bridging it — the model simply cannot look there. Architectures hedge by stacking patterns, alternating sparse and dense layers, or keeping a few full-attention heads. In practice, well-designed sparse models trade a small accuracy loss on long-range tasks for order-of-magnitude gains in speed and memory, which is usually the right trade for local deployment.

Why it matters for self-hosting

Sparse attention is one of the main reasons models can handle long contexts on modest hardware. By touching fewer token pairs, it shrinks both the compute of the prefill phase — the part you feel as time-to-first-token when you paste in a long document — and the memory footprint of the key-value cache, which is what usually exhausts VRAM long before the model weights do. For a sovereign Bitcoiner feeding node logs, transcripts, or a whole repository into a locally run model through Ollama or llama.cpp, sparse attention is often the difference between a large context window fitting on your GPU and not. It composes with quantization: one shrinks how much you store per value, the other how many values you must store at all.

The pattern to remember: attention is the budget, and sparsity is how the architecture spends it where dependencies actually live. See the prefill phase it accelerates and the broader attention mechanism it approximates.