Evaluating Retrieval-Augmented Systems
In one line: a retrieval-augmented system has two stages that fail for different reasons, and a single end-to-end score cannot tell you which one broke.
Where the failures actually are
The finding that should reorder everyone's priorities: when these systems fail, the failure is in retrieval roughly 73% of the time, not in generation. And a large share of retrieval failures trace further back still — to ingestion and chunking, before any query is ever run.
The practical consequence is blunt. Teams debug the prompt, swap the model, tune the temperature, and the problem was that the right passage was never retrieved. No reranker rescues bad chunks, and no prompt rescues a missing document.
That single question — was it in the context? — splits the debugging in two and takes seconds to answer. It is the first thing to check and the first thing to say in an interview.
Four metrics, split by stage
The standard framing separates the two stages and gives each two metrics.
| Metric | Stage | Asks |
|---|---|---|
| Context recall | Retrieval | Of what was needed to answer, how much did we retrieve? |
| Context precision | Retrieval | Of what we retrieved, how much was relevant? |
| Faithfulness | Generation | Is every claim in the answer supported by the retrieved context? |
| Answer relevance | Generation | Does the answer address the question that was asked? |
Each one isolates a different failure, and knowing which is low tells you what to fix:
Low context recall — the information was not retrieved. Look at chunking, the embedding model, and query rewriting.
Low context precision — you retrieved noise alongside the signal. The generation model now has to ignore irrelevant material, which it does imperfectly, and you are paying for those tokens. Look at the reranker and at k.
Low faithfulness — the model had the right context and made something up anyway. This is the classic hallucination case, and it is a prompting, model, or grounding-instruction problem.
Low answer relevance — a fluent, well-supported answer to a different question. Often a query-understanding problem rather than a generation one.
The precision-recall trade shows up again
Retrieve more chunks and context recall rises — you are more likely to have included what was needed. Context precision falls, because more of what you retrieved is noise.
That is the same trade from the classification lesson, and here it has two extra costs that make it sharper: every additional chunk is tokens you pay for on every request, and a longer context dilutes the model's attention, so faithfulness can fall as recall rises.
So there is an interior optimum in k, and finding it is an empirical exercise rather than a matter of taste. Saying "I'd sweep k and watch faithfulness as well as recall, because they move in opposite directions" is a precise answer where "I'd retrieve the top 5" is a guess.
Measuring retrieval without labelling everything
The obvious objection: computing context recall needs to know which passages were required, which means annotation.
Three practical routes, in increasing order of effort:
Synthesise the question from the passage. Take a chunk, have a model write a question that chunk answers, then check whether retrieval returns that chunk for that question. This generates a labelled retrieval set cheaply from a corpus you already have.
Mine real queries and label the top-k only. You do not need exhaustive relevance judgements to compute precision at k — only the retrieved items need labels.
Use the generation stage as a weak signal. If the answer is judged unfaithful and the correct information is absent from the context, that is a retrieval failure without anyone labelling anything.
Evaluate ingestion, not just query time
The last point follows from the failure distribution. If most retrieval failures originate in chunking, then chunking needs its own evaluation and almost never gets one.
Things worth measuring before any query runs: what fraction of chunks are truncated mid-sentence or mid-table, how many are so short they carry no usable meaning, whether structural elements like tables and lists survived extraction intact, and whether each chunk retains enough surrounding context to be interpretable alone.
A chunk that reads "it increased by 12% in that period" is retrievable and useless, because nothing in it says what or when. That failure is invisible to every query-time metric and is fixed at ingestion by carrying section headings into each chunk.
Key takeaway
Retrieval is the failure roughly 73% of the time, so ask "was it in the context?" before touching the prompt. Split the metrics by stage — context recall and precision for retrieval, faithfulness and answer relevance for generation — because each one points at a different fix. Retrieving more raises recall and lowers precision, and dilutes attention enough that faithfulness can fall too, so k has an interior optimum worth sweeping for. And evaluate ingestion, because a chunk that cannot be interpreted alone is invisible to every query-time metric.
Next: building an evaluation set that survives contact with production.