Definition
Prompt compression shortens the text fed to a language model by deleting or condensing tokens that contribute little information, cutting inference cost and latency while keeping the answer roughly unchanged. Unlike compressing the internal cache or context embeddings, this operates on the visible prompt itself, so it stays interpretable and works with any model behind an API or running locally.
How it works
A representative approach, LLMLingua from Microsoft Research, uses a small auxiliary language model to score tokens by importance and a budget controller to decide how aggressively to prune, achieving up to roughly twenty times compression with little measured performance loss and end-to-end speedups of several times. Remarkably, the resulting prompts can look garbled to a human yet remain effective for the target model, and a capable model can even reconstruct the original reasoning from them. Long-context variants reorder and prune retrieved passages to fight positional bias as they shrink the input.
When to use it
Prompt compression pays off when prompts are long and repetitive, such as few-shot examples, verbose retrieved documents, or chain-of-thought scaffolding. For sovereign deployments paying in their own electricity and VRAM rather than per-token API fees, the win is throughput and the ability to fit more useful content under a fixed window. The trade-off is a small accuracy risk and an extra model in the pipeline, so it is worth measuring on your own tasks before relying on it.
Compare the embedding-space alternative, context compression, and see lost in the middle for the positional effect long-context variants address.
In Simple Terms
Prompt compression shortens the text fed to a language model by deleting or condensing tokens that contribute little information, cutting inference cost and latency while…