What Retrieval Solves, and What It Does Not
In one line: if the model does not know something, that is retrieval; if it knows the material and behaves wrongly, that is not — and getting this backwards is the most expensive mistake in the area.
The distinction
A model's weights contain what it learned in training. Anything outside that — your documentation, this customer's history, yesterday's prices, an internal policy written last week — is unknown to it and cannot be prompted into existence.
Retrieval supplies that at request time. Fine-tuning changes how the model responds, not what facts it has access to.
The costly version of the mistake is fine-tuning a model on your documentation. It appears to work — the model starts using your vocabulary — and it produces a system that confidently states facts it half-remembers, with no way to correct a single one short of another training run.
Three properties that make retrieval the default
Facts stay updatable. A document changes, you re-index it, and the next answer is correct. Knowledge baked into weights goes stale and cannot be patched.
Answers become attributable. The passages that produced an answer are known, so you can cite them. That is not a nicety — in most enterprise settings it is the requirement that makes the system deployable at all, because a claim a human can check is a claim a human can accept.
Access control remains possible. Retrieval happens per request with the requester's identity available, so what a user can see is a filter. Weights have no notion of who is asking, so a fine-tuned model has permanently absorbed whatever it was trained on.
| Retrieval | Fine-tuning | |
|---|---|---|
| Adds | Knowledge | Behaviour, format, tone |
| Update cost | Re-index one document | A training run |
| Attribution | Natural — you have the sources | Impossible |
| Per-user access control | A filter at query time | Cannot be expressed |
| Cost per request | Higher — you pay for the context tokens | Lower — nothing extra in the prompt |
| Latency | Adds a retrieval step before generation | None added |
The last two rows are the honest cost. Retrieval makes every request longer and therefore more expensive, and it puts a lookup in the latency path. Those are real, and they are usually worth paying — but a candidate who presents retrieval as free is missing half the trade.
What retrieval cannot fix
Worth being explicit, because it is where designs over-reach.
Reasoning failures. If the model has the right passage and draws the wrong conclusion, more retrieval does not help. That is a model capability or a prompting problem.
Format and tone. Retrieval supplies content; it does not change how the model writes. Wanting terse, structured, on-brand output is a fine-tuning or constrained-decoding problem.
Anything requiring the whole corpus at once. "How many of our customers mentioned pricing last quarter?" is an aggregation query. Retrieval returns a handful of passages, and no amount of them answers a question about the whole set. This one is worth naming because users ask it constantly and the system will confidently answer from five documents.
The fix is not better retrieval — it is routing. Recognise aggregation intent and send it somewhere that can actually count, or refuse. A system that cannot tell a lookup from an aggregation will be wrong in the most convincing possible way.
The shape of the pipeline
Two halves, and they fail at very different rates.
The distribution is the reason this chapter is ordered the way it is: when these systems fail, retrieval is the cause roughly 73% of the time, and a large share of retrieval failures originate in ingestion — before a query has ever been run.
That inverts where most teams spend their effort. Prompt iteration is visible, immediate and satisfying; chunking is invisible, offline and dull. The next two lessons are about the dull half, because that is where the failures live.
Key takeaway
Retrieval adds knowledge and fine-tuning changes behaviour, and confusing them produces a model that confidently half-remembers your documentation with no way to correct a fact. Retrieval is the default because facts stay updatable, answers stay attributable and access control stays expressible — at the honest cost of longer, more expensive requests and a lookup in the latency path. It cannot fix reasoning, format, or any question requiring the whole corpus at once. And retrieval is roughly 73% of RAG failures, most of them originating in ingestion.
Next: getting the documents in, which is harder than it sounds.