Lexical and Semantic, Together

In one line: BM25 fails on vocabulary and embeddings fail on exactness, and those are different queries, which is why production search runs both.

The two failure sets, and they do not overlap

BM25 misses when the words differ. A document about notebooks does not match laptop. A page about automobile insurance does not match car insurance. The information is there; the terms are not.

Embeddings miss when the words must match exactly. Four cases, and they are the ones that matter most in practice:

• Identifiers. SKU-4471-B, ISBN 978-0-13, an error code. An embedding of a part number is close to embeddings of other part numbers, which is exactly wrong.
• Negation. shoes without laces embeds close to shoes with laces. The vector barely moves; the meaning inverts.
• Numeric ranges. under £80 is not a semantic property. £79 and £81 are near-identical in embedding space and one of them is disqualifying.
• Rare proper nouns. A name the embedding model never saw during training gets an uninformative vector.

The last box carries the design. Numeric ranges and identifiers are not retrieval problems at all — they are filters, and the query understanding stage should have extracted them. Asking either retrieval mode to handle under £80 semantically is asking the wrong component.

Combining two scores that are not comparable

BM25 returns an unbounded score whose scale depends on the corpus and the query. Cosine similarity returns something in a fixed range. They are not on the same scale, and no amount of tuning a weighted sum makes them so — the right weight differs per query.

The standard answer is to combine ranks rather than scores. Reciprocal rank fusion assigns each result a score based on its position in each list and sums:

RRF(d) = SUM over lists L of  1 / (k + rank of d in L)

with k a small constant, conventionally 60, that dampens the influence of the very top positions.

Why this works is worth stating: rank is scale-free. A document ranked third by BM25 and fifth by the vector index contributes the same regardless of what the raw numbers were. It also degrades gracefully — a list that returns nothing simply contributes nothing.

The cost is that you discard magnitude. A document BM25 scored overwhelmingly higher than everything else looks the same as one that barely edged out second place. Where you have the labels to calibrate scores properly, a learned combination beats RRF; where you do not, RRF is the sensible default.

When hybrid is not worth it

Worth saying, because "use hybrid retrieval" is the reflexive answer and it is sometimes wrong.

If the corpus vocabulary is controlled and the queries use the same vocabulary — an internal parts catalogue, a code search over identifiers — BM25 alone is better and much cheaper. There is no vocabulary mismatch to fix, and the embedding index adds infrastructure, an embedding model to maintain, and a one-way door on the model choice.

The signal that hybrid earns its cost is a measurable vocabulary gap: users describing things differently from how documents describe them. Reformulation chains show this directly.

The one-way door

The embedding model is a semi-permanent decision, and this is worth raising unprompted.

Changing it invalidates every stored vector, so switching means re-embedding the entire corpus. On a large index that is a substantial job, and during it your index is inconsistent — some vectors from the old model, some from the new, and distances between them are meaningless.

Two consequences for the design. Keep the source text, always, so re-embedding is possible at all. And plan the migration as a build-alongside-and-swap rather than an in-place update, because a partially re-embedded index is worse than either version.

Where the semantic side actually earns its place

Two things it does that no amount of synonym curation matches.

It generalises to phrasings nobody anticipated. A synonym list covers the mismatches you thought of; an embedding covers the ones you did not. On a long tail of unique queries that difference is large.

It handles multi-word concepts. things to do with kids on a rainy day has almost no useful term overlap with a page titled Indoor family activities. That is a query no lexical system answers and no synonym list saves.

Both of those are tail phenomena, which is a useful way to think about the split: lexical retrieval carries the head, semantic retrieval carries the tail. Head queries are short, common, and use the corpus vocabulary. Tail queries are long, unique and conversational.

What to say about it

The compact version, which covers the ground without over-claiming:

I'd run both. BM25 for exactness, identifiers and the head; a vector index for vocabulary mismatch and the conversational tail. Structured constraints — price, size, availability — go to filters rather than to either retriever, because they're hard constraints and neither retrieval mode represents them properly. Fuse on rank with RRF unless I have labels to calibrate a learned combination. And I'd keep the source text, because the embedding model is a one-way door.

Next: the hard constraints, and the filtering problem underneath them.