Quantisation and memory

The memory bill, in bytes

A float32 vector of 768 dimensions takes 3,072 bytes before an id, a tenant field or a graph edge. Ten million of them are a little over thirty gigabytes. That decides whether the collection fits on one node with room for the operating system.

Quantisation (storing each number with fewer bits) reduces the bytes per dimension. Float16 halves the bill, int8 quarters it, and binary takes one bit, which is thirty-two times smaller than float32. The saving is real and the cost is precision. The only question is how much recall a given ratio costs on your queries.

Write the maths down as bytes per vector times live vectors, then multiply by replicas and headroom. Quantisation changes the first term and nothing else. Deleted vectors and stale segments still take memory until compaction, which is why a collection can be empty of live rows and still full. A report that shows live vectors rather than stored bytes will overstate the free space by exactly the amount the compaction job has not reclaimed.

Scalar quantisation is a per-dimension promise

Scalar quantisation maps each dimension from its observed range onto 256 integer levels. The index stores a scale and an offset per dimension, encodes a vector as one byte per dimension, and decodes it on the fly. It is the least invasive option: four times smaller, usually within a point or two of float32 recall. The encoder is cheap enough to run on every query, which is why the approach is popular.

The range is the weak point. A single outlier stretches the range, and every normal value is then squeezed into a handful of buckets. Clip the range at a high percentile (a cutoff near the top of the values). Keep the raw float vectors for the few values beyond it. Measure the reconstruction error (how far the rebuilt vector is from the original) rather than trusting the default.

The query has to be quantised with the same range as the corpus, or the distances compare two different coordinate systems. That link is easy to forget in a service that loads a new index version. The vectors and the query encoder must ship together, as one package with one version number.

Four bytes to one, then thirty-two to one

import faiss

scalar = faiss.IndexScalarQuantizer(dim, faiss.ScalarQuantizer.QT_8bit)
scalar.train(vectors)
scalar.add(vectors)
distances, ids = scalar.search(queries, 10)

quantizer = faiss.IndexFlatIP(dim)
pq = faiss.IndexIVFPQ(quantizer, dim, 256, 32, 8)  # 32 subspaces, 8 bits each
pq.train(vectors)
pq.add(vectors)
pq.nprobe = 16
distances, ids = pq.search(queries, 10)

Product quantisation splits the vector

Product quantisation splits a vector into subvectors. It builds a small codebook (a set of representative points) for each subspace, and stores the id of the nearest centroid instead of the values. Thirty-two subspaces with 256 centroids each turn a 768-dimension float32 vector into 32 bytes, a ninety-six-fold reduction.

Search does not decode the vector. It works out the distance from the query to every centroid in every subspace, then sums the table entries named by the stored codes. That lookup is the reason product quantisation is fast. It is also the reason its error is bounded by how well the codebooks describe the data.

Codebooks are trained, so they reflect the training set. A corpus that drifts away from the sample used at training time loses recall without a warning, and a new embedding model needs new codebooks. Keep the training sample, version the codebooks, and retrain on a schedule rather than only at build time.

Choose by measurement, not by ratio

Measure three options on the same queries: float32, int8 and a product-quantised index. For each, record recall@10 against exact search, bytes per vector and p95 latency. The cheapest option that meets the recall target wins; the ratio on the box is not the deciding number. Use the same query set for every option, not a fresh random sample, because the tail is where quantisation fails.

Binary quantisation is the extreme option: one bit per dimension, Hamming distance (the number of differing bits), thirty-two times smaller than float32. It is only usable when the embedding model was trained for it, or when a float rerank of the top thousand saves the recall. Without that rerank, it is a fast index with a poor answer.

Quantisation belongs to the collection and to the embedding model, not to the product. Record the encoder, the codebook version and the range in the collection metadata, and re-quantise in a background job when any of them changes. Cut over only after the recall sample agrees.