KV Cache Quantization

KV cache quantization reduces the memory footprint of long-context inference by storing cached attention keys and values in low-precision integer formats instead of 16-bit floats. Because the cache grows linearly with sequence length, a long prompt on a large model can consume more memory than the model weights themselves; at extreme context lengths the cache alone can run into hundreds of gigabytes, exceeding any single GPU. Quantizing it is one of the most direct ways to make long windows fit on modest hardware.

How it is done well

Naive uniform quantization hurts accuracy because the key and value tensors have very different statistics. The KIVI method, presented at ICML 2024, showed that keys are best quantized per channel while values are best quantized per token, and that with this asymmetric scheme a 2-bit cache preserves generation quality while cutting peak memory roughly in half and enabling far larger batch sizes and higher throughput. Other schemes isolate rare outlier values into a small high-precision component so the bulk can go to sub-4-bit safely.

Why it matters for sovereign inference

For someone serving models on owned hardware rather than renting cloud GPUs, cache quantization is what turns a 128K-token window from a theoretical spec into something that runs on a single consumer card. It trades a small, often negligible quality loss for a large reduction in VRAM, which can be the difference between a model that loads and one that does not. It composes with weight quantization and with attention-efficiency tricks to compound the savings.

See the KV cache it compresses and long context window for the memory pressure it relieves.