Vector Quantization

Vector quantization (VQ) is a lossy compression technique, borrowed from signal processing, that maps any vector in a high-dimensional space to the nearest member of a finite, pre-learned set of representative vectors. That set is called a codebook, each entry in it is a codevector or centroid, and a compressed vector is stored as nothing more than the integer index of its closest centroid. Instead of keeping thousands of floating-point numbers per embedding, you keep one small ID and look the rest up. For anyone running retrieval or semantic search on their own hardware, VQ is the base concept behind almost every trick that makes a large vector index fit in the RAM you actually own.

How a codebook is learned

The standard way to build a VQ codebook is k-means clustering, which is itself a vector-quantization algorithm. Given a target codebook size k, k-means partitions the training vectors into k clusters and sets each centroid to the mean of its cluster, iterating until the centroids stabilize and the total distortion — the summed squared distance from each vector to its assigned centroid — stops shrinking. At encode time, a new vector is assigned the index of its nearest centroid; at decode time, that index is replaced by the centroid itself, an approximation of the original. The quality of the approximation depends on how well the codebook covers the data distribution: more centroids mean lower distortion but a bigger codebook and slower nearest-centroid search.

The compression arithmetic

The payoff is easiest to see with numbers. A 1,024-dimension embedding stored as 32-bit floats costs 4,096 bytes. Quantized against a codebook of 256 centroids, the same vector is stored as a single 8-bit index — one byte — plus a shared codebook amortized across the whole collection. The catch is that one global codebook is far too coarse for high-dimensional data: 256 points cannot meaningfully tile a 1,024-dimensional space. That is why practical systems extend the idea rather than use it raw. Product quantization splits the vector into sub-vectors and quantizes each piece with its own small codebook, multiplying the effective resolution. Binary quantization is VQ taken to the one-bit-per-dimension extreme. Scalar quantization approaches, like the int8 and int4 schemes used in LLM weight quantization, apply the same keep-less-precision philosophy per value rather than per vector.

Why it matters for self-hosted AI

Sovereign AI lives or dies on memory. A million embeddings at full float32 precision is roughly 4 GB before the index structure is even counted; on a home server that competes with the model weights themselves for RAM and memory bandwidth. Quantized indexes shrink that footprint by an order of magnitude or more, which is frequently the difference between a retrieval-augmented setup running comfortably on hardware in your rack and one that quietly demands a cloud vector database — and with it, a third party watching your queries. The same logic that leads a node runner to keep their own copy of the chain leads a self-hoster to keep their own index, and VQ is what makes that index affordable.

Nothing is free: quantization discards information, so recall drops slightly, and most production systems compensate by re-scoring a shortlist of candidates with the original full-precision vectors. Understanding plain VQ — codebook, centroid, index — makes every one of those derivative schemes easy to reason about, because they are all the same move: trade a little geometric precision for a lot of memory, and keep the compute at inference time on machines you control.

The idea has escaped search, too. Modern neural codecs quantize audio and images into sequences of codebook indices, and residual vector quantization — quantizing the leftover error with a second codebook, then a third — recovers precision without an impossibly large single codebook. Wherever you see "tokens" standing in for continuous signals, a learned VQ codebook is usually doing the translation.