Choosing an Embedding Model

In one line: changing the embedding model invalidates every vector you hold, so this is the closest thing in the stack to a one-way door.

Why it is semi-permanent

Vectors from two different models are not comparable. They occupy different spaces with different geometries, and a distance between them is meaningless.

So switching models means re-embedding the entire corpus, rebuilding the index, and re-validating quality — hours to days of compute on a large corpus, plus a migration during which two indexes exist.

The design consequence: budget the re-embed cost when you choose, and store the raw text. A corpus you can re-embed is recoverable; one where you kept only vectors is not. That sounds obvious and is a real failure — teams discard source text after ingestion and discover they cannot ever change models.

Dimensions are a permanent multiplier

From the estimation chapter: storage and index memory scale linearly with dimensions. Moving from 768 to 1,536 doubles the raw vectors, doubles the index, and roughly doubles the memory bill forever.

The quality gain is often small. Larger models are usually better, and the margin between a good 768-dimension model and a good 1,536-dimension one is frequently a few points of recall — which may not be worth doubling the infrastructure.

Context window versus chunk size

An embedding model has a maximum input length, and text beyond it is silently truncated — no error, just a vector representing the first part of a longer document.

Two consequences that interact with chunking:

Your chunk size must fit the model's window, with room for any prefix or instruction you prepend.

A longer window is not automatically better. Embedding a very long passage into one fixed-length vector averages away its specifics — a 4,000-token chunk becomes a vector about the general topic and stops matching precise questions. Smaller chunks retrieve more precisely and carry less context.

That tension is the real chunking trade-off and it has no universal answer, only a measurement.

Domain fit beats general quality

A general model is trained on broad web-scale similarity, and its notion of relatedness may not match a specialist one. Legal citation relevance, clinical equivalence, code that solves the same problem — these are relations the model may simply never have been asked to represent.

The fine-tuning row carries a hidden operational cost worth naming: every time you retrain the embedding model, you re-embed everything. A fine-tuned embedding is not a model you iterate on weekly.

How to actually choose

Public leaderboards are a starting filter and a weak signal, for a reason worth stating: they measure average performance across many tasks, and you have one task. A model ranked fifteenth overall can be first on yours.

The approach that works:

A hundred pairs is a small enough set to assemble in a day and large enough to separate models that differ meaningfully. The effort is trivial next to the cost of choosing wrong.

Hosted or self-hosted

Embedding is one of the cheapest model workloads, so the calculation differs from generation.

A hosted API is simple and costs per token, with the corpus embed as a one-off spike and queries as steady low volume. Self-hosting an embedding model is unusually tractable — they are small, they batch extremely well, and they have no autoregressive decode phase, so a single GPU handles very high throughput.

The point that usually decides it: query latency sits in the request path. An external API call adds tens of milliseconds to every search, and if your retrieval budget is 50ms that may not fit. Self-hosting is often chosen for latency rather than cost.

Next: how nearness is actually measured.