Context Compression

Context compression packs a long input into a much smaller set of learned representations — often called memory slots or soft tokens — that a model conditions on as if they were the original text. Unlike retrieval, which fetches relevant chunks from an external store, context compression aims to preserve the meaning of the whole input while drastically reducing the number of vectors the model must attend over. The payoff is direct: attention cost grows with sequence length, so fewer effective tokens means lower latency, smaller memory footprints, and longer documents fitting into the same context window.

How it works

Methods such as the In-Context Autoencoder (ICAE) train an encoder — frequently the target model itself, adapted with a lightweight fine-tuning method like LoRA — to read a long context and emit a small number of dense slots that the decoder can then expand or reason over. Reported compression ratios around four-to-one are typical for this family, with the slots directly consumable by the target model without retraining it from scratch. Recursive variants chain the process, folding already-compressed memories together so that very long inputs collapse into a fixed-size representation. Related techniques include gist tokens (training a model to distill an instruction into a few special tokens) and architectural approaches that maintain compressed memory across segments; the shared idea is always the same trade — spend a one-time encoding pass to make every subsequent use of that context cheap.

Compression versus prompt pruning

Context compression is the embedding-space cousin of prompt compression, which instead drops or rewrites actual text tokens. Soft-token compression can be denser and more faithful because it is not constrained to human vocabulary, but it produces representations that are not human-readable, are tied to the model that learned them, and usually require a training step to set up. Text-level pruning stays interpretable, auditable, and model-agnostic at the cost of coarser compression. The honest engineering guidance: compression is lossy either way — details can vanish in the squeeze — so verify on your own tasks that what survives compression is what your application actually needs, especially for precise recall like exact figures, addresses, or code.

Not the same as KV-cache tricks

Context compression is easily confused with a neighbouring family of optimizations that squeeze the model's runtime state: KV-cache quantization, eviction policies that drop attention entries for less-relevant tokens, and similar inference-engine techniques. Those operate transparently on whatever tokens you feed in, trading precision for memory at serving time. Context compression happens earlier and more deliberately — it changes what the model is given, replacing thousands of tokens with a learned digest. The two compose: a compressed context still benefits from an efficient cache underneath it, and self-hosters chasing long documents on small GPUs typically end up using both.

Where it pays off for self-hosters

For an operator running local models, context compression is most attractive when the same long material is reused across many queries: documentation, a codebase, machine manuals, historical logs. The dense memory can be computed once and cached on disk, then loaded for pennies of compute per query — effectively pre-paying the reading cost. On consumer GPUs where VRAM is the binding constraint, a four-fold reduction in effective context is the difference between a long context window you can actually run and one you can only read about. It also composes with retrieval rather than competing with it: RAG narrows which documents enter the window, and compression shrinks what they cost once inside. Keeping the whole loop — encoder, cache, and model — on your own hardware means the corpus being compressed never leaves your custody, which is the point of hosting it yourself in the first place.