Prompt Compression

Prompt compression shortens the text fed to a language model by deleting or condensing tokens that contribute little information, cutting inference cost and latency while keeping the answer roughly unchanged. Unlike compressing the internal KV cache or context embeddings, this operates on the visible prompt itself, so it stays interpretable, requires no access to model internals, and works with any model behind an API or running locally. The insight it exploits is that natural language is highly redundant: a large fraction of tokens in a typical prompt exist for human readability, not for the model's benefit.

How it works

A representative approach, LLMLingua from Microsoft Research, uses a small auxiliary language model to score tokens by how surprising — and therefore how informative — they are, plus a budget controller that decides how aggressively to prune different parts of the prompt. Instructions and questions are usually preserved nearly verbatim, while few-shot examples and retrieved documents absorb most of the cuts. Published results report compression up to roughly twenty times with little measured performance loss on reasoning and summarization benchmarks, along with end-to-end speedups of several times. Remarkably, the compressed prompts can look garbled to a human yet remain effective for the target model, and a capable model can even reconstruct much of the original reasoning from them. Long-context variants extend the idea by reordering and pruning retrieved passages, which also fights the positional bias where models attend poorly to material buried mid-context.

When to use it

Prompt compression pays off when prompts are long and repetitive: stacks of few-shot examples, verbose retrieved documents in a RAG pipeline, sprawling conversation histories, or chain-of-thought scaffolding that repeats boilerplate. It pays off least on short, dense prompts where every token is already doing work — compressing a two-sentence question buys nothing and risks mangling it. The trade-offs are real: a small accuracy risk that varies by task, an extra model in the pipeline adding its own latency and memory footprint, and reduced debuggability, since the prompt the model actually saw no longer reads cleanly. The honest engineering posture is to measure on your own tasks — run your evaluation set with and without compression at several ratios, and let the numbers decide rather than the benchmark headlines.

The sovereign angle

For self-hosted deployments paying in electricity and VRAM rather than per-token API fees, the economics shift but do not disappear. The win becomes throughput and context headroom: a compressed prompt means more requests per second from the same GPU, and more useful content fitted under a fixed context window — often the binding constraint on consumer hardware running quantized open-weight models. A homelab running a local assistant over its own document store feels this directly: compression can be the difference between retrieval results fitting in context or being truncated. There is also a quiet privacy benefit to the local version: cloud compression services see your prompts by definition, while a small scoring model running beside your main model keeps the entire pipeline on hardware you control — the same logic that favors a self-hosted stack in the first place. Anyone running inference alongside a mining operation, where the power budget is already accounted for, gets the compute savings essentially free.

Compare the embedding-space alternative, context compression, which achieves larger ratios by giving up human readability entirely, and see lost in the middle for the positional effect that long-context compression variants are designed to address. As with most optimizations in a local stack, the discipline is the one miners already apply to tuning: change one variable, measure honestly, and keep the configuration only if the numbers earn it.