Embedding and Vector Index Sizing

In one line: vectors are cheap and the index over them is not, so the number that decides your retrieval architecture is raw size multiplied by an overhead factor you should know.

Raw storage

Raw bytes  =  vectors  x  dimensions  x  bytes per value

At float32 that is 4 bytes per dimension. Some anchors:

The dimension term is linear and easy to forget: moving from a 768-dimension model to a 1,536-dimension one doubles your storage and your memory bill for a retrieval quality gain that is often small. Choosing the embedding model is a capacity decision, not only a quality one, and that is a good thing to say out loud.

Chunks, not documents

A common estimation error is counting documents. You embed chunks, and one document produces many.

1,000,000 documents
  x ~10 pages each
  x ~2 chunks per page
  = 20,000,000 chunks
  x 768 dims x 4 bytes
  = ~61 GB raw

A twenty-fold multiplier hides in "one million documents". Ask for the chunking strategy before estimating, because chunk size is what converts a document count into a vector count — and halving chunk size doubles everything downstream.

The index adds overhead on top

Raw vectors are not searchable at speed. A graph-based index stores neighbour links per vector, and those links are not free.

HNSW total  ~=  raw vectors  x  1.5 to 2

So the 61GB above becomes roughly 90–120GB once indexed, and the practical question becomes whether that fits in RAM on machines you are willing to pay for.

The quantisation ladder

Compressing the vectors themselves is the main lever, and the ladder is worth knowing because the steps differ enormously in both saving and cost.

Scalar quantisation to int8 is the default move. Roughly 4× smaller for a small recall cost, and it turns a 120GB index into about 30GB, which is the difference between a large specialised machine and an ordinary one.

The aggressive end of the ladder is only sane in combination with a rerank stage: search a heavily compressed index to get a generous candidate set, then rescore those few hundred candidates against full-precision vectors. That is the cascade pattern again, applied to retrieval — cheap and approximate first, exact and expensive on the survivors.

Where in-memory stops being economical

There is a scale at which keeping everything in RAM stops making sense, and it lands somewhere around a hundred million vectors for typical dimensions.

Past that, the practical path is disk-based indexing: the graph lives on NVMe with the hot portion cached in memory. The trade is latency — a disk-resident index answers in tens of milliseconds rather than single digits — and the saving is that you are no longer paying for terabytes of RAM.

The middle branch is the one candidates skip and it is often the right answer. Most corpora partition naturally — by customer, by locale, by recency — and a query usually only needs one partition. Sharding by tenant turns one impossible index into a thousand easy ones, and it gives you data isolation for free.

Index build time and freshness

The number people forget entirely. Building a graph index over a hundred million vectors is hours, not minutes, and it is CPU-intensive.

That has a direct architectural consequence: you cannot rebuild the index every time a document changes. The usual resolution is a two-tier arrangement — a large, periodically rebuilt main index plus a small, frequently updated delta index, with queries hitting both and merging results. The delta is folded into the main index on the next rebuild.

Putting it together

20,000,000 chunks x 768 dims x 4 bytes  =  ~61 GB raw
x 1.75 for HNSW graph overhead          =  ~107 GB in memory
quantised to int8                       =  ~27 GB
  -> fits comfortably on one large machine, with a replica for availability
  -> rebuild nightly, delta index for the day's changes

Then state the sensitivity, as always: at 1,536 dimensions this doubles; at half the chunk size it doubles again; and past roughly ten times this corpus, sharding or disk-based indexing becomes the conversation.

Next: what training and fine-tuning actually cost, and when to do neither.