Hybrid Search and Rank Fusion

In one line: dense and lexical retrieval fail on different queries, so running both and merging is the production standard — and merging them correctly means ignoring their scores.

What lexical retrieval is still better at

BM25 scores a document by term overlap with the query, weighted by how rare each term is and normalised for document length. It has no notion of meaning at all, and that is precisely its advantage on a specific class of query.

The pattern is clean: dense generalises, lexical is exact. A term the embedding model never saw in training is meaningless to it and perfectly ordinary to BM25, which is why new product names, error codes and internal acronyms are a systematic dense-retrieval weakness.

That also explains why the two are complementary rather than redundant. They are not two attempts at the same thing with different accuracy — they fail on disjoint query sets.

Why you cannot just add the scores

The obvious merge is to normalise both scores and take a weighted sum. It is the intuitive approach and it works badly, for a reason worth being able to state.

The two scales are incompatible. BM25 is unbounded and positive, with values depending on corpus statistics, document length and term rarity. Cosine similarity is bounded to −1 to 1. There is no principled mapping between them.

Worse, both distributions shift per query. A query with rare terms produces much larger BM25 values than a common-term query, so a normalisation fitted globally is wrong for most individual queries. And in a sharded index, computing a global normalisation requires a coordination step you would rather not have.

Reciprocal rank fusion

The standard solution discards the scores entirely and uses only position.

RRF score(d)  =  sum over rankers of  1 / (k + rank(d))

k is a constant, conventionally 60

Each ranker contributes based on where it placed the document. A document ranked 3 by both contributes 1/63 + 1/63. One ranked 1 by a single ranker and absent from the other contributes 1/61.

Three properties make it the default:

It is normalisation-free. No knowledge of either score distribution is required, so it works across rankers, corpora and shards without tuning.

It rewards consensus over confidence. A document both rankers place highly beats one that a single ranker loves — which is exactly the behaviour you want, since agreement between two systems that fail differently is strong evidence.

The constant dampens the top. k = 60 keeps rank 1 from dominating rank 2 the way raw reciprocal rank would, so a single confident ranker cannot override the other entirely.

Where fusion sits in the pipeline

Fusion is the middle stage, not the last one.

The reranker is doing something neither retriever can, and it is worth being precise about why. Both retrievers are bi-encoders: they embed query and document independently, so no feature can express their interaction. A cross-encoder reads the pair jointly and can distinguish a passage about the topic from one that answers the question — the fourth limitation from the first lesson.

That is also why it cannot be the retriever. Scoring a pair jointly means one model pass per document, which is affordable over a hundred candidates and impossible over ten million. Retrieve wide and cheap, rerank narrow and expensive — the cascade again.

Next: keeping the index correct as the corpus changes.