Retrieval and Reranking
In one line: retrieve wide and cheap, rerank narrow and expensive, and remember that the number you pass to the model trades recall against precision against attention all at the same time.
The standard shape
Hybrid retrieval and rank fusion were covered in the vector search chapter, and the short version is that dense and lexical fail on disjoint query sets — dense generalises across vocabulary, lexical is exact on identifiers and rare terms — so the production standard runs both and fuses on ranks rather than incompatible scores.
What is new here is the reranker.
Why the reranker is a different kind of model
Retrieval uses a bi-encoder: query and document are embedded independently, so their vectors can be precomputed and compared with a distance. That independence is exactly what makes indexing possible — and it means no feature can express the interaction between a specific query and a specific document.
A cross-encoder reads the pair together and outputs a relevance score. It can notice that a passage is about the right topic and answers a different question, which the retrieval lesson identified as the fourth thing embeddings cannot do.
| Bi-encoder (retrieval) | Cross-encoder (rerank) | |
|---|---|---|
| Input | Query and document separately | The pair, together |
| Precomputable | Yes — documents embedded offline | No — one pass per pair, at query time |
| Cost | One embedding plus an index lookup | One model pass per candidate |
| Scales to | Millions of documents | Tens to low hundreds |
| Sees | Topical similarity | Whether this passage answers this question |
The "precomputable" row is the whole reason for two stages. A cross-encoder cannot be the retriever because scoring ten million pairs per query is impossible; a bi-encoder cannot be the final judge because it never sees the pair. Each does the job the other cannot.
What k trades
The number of chunks passed to the model looks like one dial and moves three things in different directions.
That fourth effect is the one that surprises people and it is the reason "just retrieve more" is not a strategy. Adding irrelevant passages does not merely waste tokens — it makes the model less likely to use the relevant one, so measured faithfulness can decline as recall improves.
So there is an interior optimum, and finding it is empirical:
Sweep k and plot both context recall and faithfulness. They move in opposite directions, and the useful setting is where their combination peaks rather than where either does.
Typical landing zones are 3–10 chunks after reranking, and the number matters less than having derived it. "I'd sweep k and watch faithfulness as well as recall, because they trade" is a much stronger answer than any specific number.
Retrieve wide, keep narrow
The two ks are different and conflating them is a common error.
Retrieval k — how many candidates the retrievers return, before reranking. This should be generous: 50 to 200. Anything dropped here is unrecoverable, because the reranker only ever sees what it was handed. This is the recall ceiling from the vector search chapter.
Final k — how many survive to the model. This should be tight, for the attention reason above.
The reranker's whole job is to make that gap safe: retrieve widely without paying for width in the prompt.
When to skip the reranker
Being fair to the alternative, because adding one is not free — it is a model call in the latency path and a component to operate.
Skip it when the corpus is small enough that retrieval is already precise, when the latency budget genuinely cannot absorb it, or when measurement shows retrieval precision is already high and the reranker is reordering things that were all relevant.
Add it when retrieval returns topically-right and unhelpful passages, which is the specific symptom it fixes. Shipping hybrid retrieval first and adding a reranker when the evaluation demands it is a defensible sequence and a good thing to say.
Metadata filtering belongs here
Retrieval in a real system is almost never unconstrained: this tenant, this language, documents this user may see, published after some date.
The vector search chapter covered why this is hard — a selective filter fragments graph traversal and collapses recall below about 1% selectivity — and the RAG-specific point is that permissions are usually the filter, which makes it a correctness requirement rather than a relevance one. That gets its own lesson.
Key takeaway
Retrieval uses a bi-encoder whose independence makes indexing possible and makes query-document interaction invisible; a cross-encoder reads the pair and can tell "about it" from "answers it", which is why the two stages exist and why neither can do the other's job. Retrieve generously — 50 to 200 candidates, since anything dropped is unrecoverable — and keep the final set tight, because raising k trades recall against precision, cost, and attention dilution that can lower faithfulness even when the right passage is present. Sweep k against both recall and faithfulness, and check retrieval recall before tuning anything downstream of it.
Next: how the surviving chunks are arranged in the prompt.