IVF and Quantisation
In one line: IVF cuts the space into regions and searches a few of them, and quantisation shrinks the vectors themselves — and of the two, compression usually has the bigger effect on what you can afford.
IVF: partition, then probe
Build time clusters the vectors into nlist partitions, each with a centroid. Query time compares the query against the centroids, picks the nprobe closest partitions, and searches exhaustively inside only those.
The saving is the ratio: search nprobe partitions out of nlist, so you examine roughly nprobe / nlist of the corpus.
The two parameters
nlist — how many partitions. Set at build time. A common starting point is around the square root of the vector count. More partitions mean smaller ones and cheaper scans, and also that each holds fewer neighbours so you need to probe more of them.
nprobe — how many partitions to search. Set at query time, and therefore the operating dial, exactly like efSearch. Raising it improves recall and costs latency, and it does not affect memory at all.
| nlist | nprobe | |
|---|---|---|
| Set at | Build time — permanent | Query time — free to change |
| Raising it | Smaller partitions, faster scans, more probes needed | Better recall, slower query |
| Memory effect | Marginal | None |
| Typical | Around sqrt(N) | 1–10% of nlist, tuned to recall |
The edge-of-cluster problem
IVF's characteristic failure, and worth being able to explain because it is the reason its recall trails HNSW at equal speed.
A query landing near a partition boundary has true neighbours sitting just across it. If that neighbouring partition is not among the nprobe searched, those results are lost — and the query looks entirely successful.
The mitigation is a larger nprobe, which is exactly the recall-for-latency trade, and it applies to every query rather than only the boundary ones. That is the structural reason graph indexes tend to win on recall-per-millisecond: a graph has no partition walls to fall off.
Quantisation matters more than the index choice
Whichever structure you pick, the vectors themselves usually dominate memory. Compressing them is the highest-leverage decision available.
| Method | Size versus float32 | Recall cost | Notes |
|---|---|---|---|
| float16 | 1/2 | Negligible | Nearly free; do it by default |
| Scalar (int8) | ~1/4 | Small | The standard first move |
| Product quantisation | 1/10 or less | Real | Needs a rescore stage to be usable |
| Binary | 1/32 | Large alone | Only viable as a first stage |
Scalar quantisation to int8 is the default. Roughly four times smaller for a small recall cost, and it turns a 120GB index into about 30GB — frequently the difference between a specialised machine and an ordinary one.
Product quantisation, and why it needs a rescore
Product quantisation splits each vector into sub-vectors, clusters each sub-space separately, and stores the cluster ids rather than the values. Compression is dramatic and the stored vector is genuinely lossy — distances computed against it are approximations of approximations.
Used alone, recall suffers noticeably. Used properly, it is the first stage of a cascade:
That is the cascade pattern from the estimation chapter appearing inside retrieval. The compressed index does the wide, cheap sweep; full-precision vectors do the narrow, exact scoring. It recovers most of the precision compression gave away, and it requires keeping the full vectors somewhere — often on disk, since only a few hundred are read per query.
Being able to say "product quantisation with a full-precision rescore" rather than just "quantise" is the difference between naming a technique and describing a design.
Choosing between them
The middle branch is the one candidates skip and it is often correct. Most corpora partition naturally and most queries touch one partition, which turns one impossible index into many easy ones — and delivers tenant isolation as a side effect.
Key takeaway
IVF clusters the space and searches nprobe of nlist partitions, with nprobe as the free query-time dial and nlist fixed at build. Its characteristic failure is the query near a partition boundary whose neighbours sit just across it, which is why graph indexes lead on recall-per-millisecond — a graph has no walls. But the bigger lever is usually quantisation: int8 as the default four-fold saving, and product quantisation only alongside a full-precision rescore, which is the cascade pattern applied inside retrieval.
Next: the thing that breaks all of this — adding a filter.