Evaluating the Pipeline by Stage
In one line: this pipeline has six places to fail and one output, so a single quality number tells you something is wrong and never what.
The first question
Before any metric, one binary check splits the problem in half and takes seconds.
Most teams debug the prompt first. The distribution says the prompt is the less likely culprit by a wide margin, and the check that would have told them costs one lookup.
Metrics per stage
| Stage | Metric | A low score means |
|---|---|---|
| Parsing | Extraction spot-checks, table integrity | The content was never usable — nothing downstream can fix it |
| Chunking | Fraction of chunks interpretable alone | Chunks embed as noise; add context or re-chunk |
| Query transform | Standalone-query accuracy on follow-ups | Follow-up turns retrieve the wrong subject |
| Retrieval | Context recall at the retrieval k | A hard ceiling on everything after it |
| Reranking | Context precision at the final k | Noise reaching the model; attention dilutes |
| Generation | Faithfulness, answer relevance | The model had it and ignored or embellished it |
The first two rows are the ones nobody measures, and they sit upstream of everything. A chunk that cannot be interpreted alone is invisible to every query-time metric — it simply never gets retrieved, and the retrieval metrics record a miss without saying why.
Retrieval recall is the ceiling. If recall at the retrieval k is 60%, no reranker, prompt or model raises final quality past 60%. Measure it before tuning anything downstream.
Reasonable targets
Useful as a starting frame rather than universal truth:
Faithfulness > 0.90 claims supported by the context Answer relevance > 0.85 the answer addresses the question Context precision > 0.80 retrieved chunks were relevant Context recall > 0.80 what was needed was retrieved
Two notes. Faithfulness is highest because an unsupported claim is the failure that destroys trust — the others degrade the experience, that one makes the system unusable. And these are measured with a model judge, which carries the biases from the evaluation chapter, so they need periodic human validation.
Building a retrieval eval set cheaply
The objection is real: context recall needs to know which passages were required, which means annotation.
Three routes, increasing in effort and honesty:
Synthesise questions from passages. Take a chunk, have a model write a question it answers, then check whether retrieval returns that chunk for that question. This produces a labelled retrieval set from a corpus you already have, at scale.
Its limitation is worth stating: generated questions reuse the passage's vocabulary, so lexical overlap is artificially high and the system looks better than it is. Pair it with real queries to calibrate the gap.
Label only the top-k of real queries. Precision at k needs judgements only on what was retrieved, so a few hundred real queries with their top-10 labelled is a day's work and much more honest.
Mine the failures you already have. Every escalation, thumbs-down and support ticket about a wrong answer is a labelled hard case. These are the highest-value examples in the system and they arrive free.
The diagnostic, in order
When quality is poor, work the pipeline backwards from where the failures actually are rather than forwards from where debugging is easiest.
1. Read the chunks. Sample the extracted text. Mangled tables, interleaved columns, chunks of pure navigation furniture. This is the highest-return hour available and it is almost always skipped.
2. Check retrieval recall. If the needed passage is not in the candidate set, nothing downstream matters.
3. Check the query. In a conversational product, is the follow-up being rewritten into standalone form?
4. Check precision and ordering. Is the right passage present but buried in the middle of the context?
5. Only then, the prompt. Grounding instructions, citation requirements, refusal behaviour.
Monitor in production, not only offline
Offline sets go stale as the corpus and the queries move. Three signals available from live traffic without labels:
Retrieval score distribution. A shift means the query mix or the corpus changed.
Refusal rate. Rising means retrieval is returning less useful context, before anyone complains.
Reformulation rate. A user immediately rephrasing is telling you the first answer failed, with no annotation required.
All three are the tier-2 and tier-3 monitoring from the foundations module, applied to this pipeline — cheap, label-free, and they move before quality metrics do.
Key takeaway
Ask "was the needed passage in the context?" before anything else — it splits six stages into two halves in one lookup, and the answer is retrieval roughly 73% of the time. Measure per stage, remembering that parsing and chunking sit upstream of every query-time metric and that retrieval recall is a hard ceiling on final quality. Build the eval set from synthesised questions for scale, real labelled top-k for honesty, and mined failures for hard cases — reported separately, since synthesised questions always flatter. And debug in the order the failures actually occur, which is the reverse of the order that is convenient.
Next: the walkthrough, where the whole pipeline gets designed live.