The RAG Pipeline

In one line: RAG is the component the design leans on hardest, and the chapter describes what it does without describing how it fails. Most production RAG problems are retrieval problems, not generation problems.

The two-stage shape is the point: retrieve widely and cheaply, then rerank narrowly and expensively. A bi-encoder can search millions; a cross-encoder can only score dozens, and is much better at it.

The pipeline

The retrieval service converts the query into a vector embedding, searches the vector database for semantically similar documentation chunks, ranks results by relevance, and injects the top-k results into the LLM prompt as grounding context.

Semantic search is why this works where keyword search would not

The step worth understanding is the first one: the query becomes a vector, and so did every document chunk.

That matters because support queries and support documentation almost never share vocabulary:

User:     "my thing won't charge"
Document: "Troubleshooting: device fails to power on when connected
           to an AC adapter"

KEYWORD search:  overlap is roughly zero -> no match
SEMANTIC search: both embed to nearby points in vector space -> match

An embedding maps text to a point in a high-dimensional space where proximity means similarity of meaning rather than similarity of characters. So retrieval works on paraphrase, synonym, and colloquialism — exactly the register real customers write in.

This is a genuinely different retrieval model from anything else in this module. The typeahead chapter's trie matched prefixes — a structural property of the string. Here the match is on meaning, which is why the index is a vector database rather than a text index.

When users and documents describe the same thing in different words, you need similarity of meaning rather than similarity of tokens — and that is the entire reason a vector database is in this design.

Worth noting the standard refinement: production systems often use hybrid search, combining semantic similarity with keyword matching. Semantic search is weak on exact identifiers — order numbers, SKUs, error codes — where the literal string is what matters and its embedding is meaningless.

Chunking

Chunking is mentioned once and it determines retrieval quality more than the embedding model does

The vector database "stores chunked and embedded product documentation." That is the only mention, and chunking is where most RAG systems are won or lost.

You cannot embed a whole document — embeddings represent a bounded amount of text, and retrieving a 40-page manual to answer one question wastes the entire context budget. So documents are split. How you split them determines what can be retrieved.

TOO SMALL (a sentence):
  "Refunds are processed within 5 business days."
  -> retrieved without its conditions. Which refunds? After what?
  -> the model answers confidently and incompletely

TOO LARGE (a whole page):
  -> the relevant sentence is buried among irrelevant text
  -> dilutes the embedding, so the chunk matches nothing well
  -> burns context budget, which Lesson 3 showed drives TTFT

The techniques that matter:

Overlap. Adjacent chunks share a boundary region, so a fact that straddles a split is retrievable from either side.

Semantic boundaries rather than fixed sizes. Split on sections, headings, and paragraphs — not every N characters. A policy split mid-sentence is a chunk that means nothing.

Attach metadata to each chunk — source document, section heading, last-updated date, product version. Retrieval can then filter as well as rank ("only chunks for the product this user owns"), which is often more valuable than better ranking.

Chunk size is a tuning parameter with more impact on answer quality than the choice of embedding model, and it is the first thing to examine when a RAG system gives confidently incomplete answers.

What RAG fixes, and what it does not

'Preventing hallucination' overstates it — retrieval failure produces hallucination that looks identical

The compliance table claims RAG "prevents hallucination." The introduction was more careful — "significantly reduces" — and the gap is the important part.

Three failure modes survive retrieval, and none of them produces a visible error:

Retrieval failure. The relevant chunk is not returned — vocabulary mismatch, bad chunking, the document does not exist, or top-k cut it off. The model then answers from parametric memory, in exactly the same fluent, confident register:

Retrieval returns 5 irrelevant chunks
  -> the prompt says "answer using this context"
  -> the context doesn't contain the answer
  -> the model answers anyway, from what it learned in training
  -> the response looks identical to a grounded one

There is no signal distinguishing a grounded answer from an ungrounded one. That is the core problem, and it is what citations exist to fix.

Ignored context. The chunk is retrieved and the model contradicts it — more common when retrieved content conflicts with strongly-held parametric knowledge, or when the instruction to ground is buried in a long prompt.

Stale index. The document changed and the embedding did not. The retrieval is confident and the content is out of date, which is precisely the failure RAG was adopted to prevent.

Grounding is a prompt instruction, not an architectural constraint — nothing in the pipeline prevents a sentence unsupported by the retrieved chunks. Designing as though RAG eliminated hallucination is the most common error in LLM system design.

Citations are the missing mechanism, and they solve three problems at once

The design never mentions attribution, and adding it is the highest-value change to the RAG pipeline.

Require the model to cite which retrieved chunk supports each claim, and return those citations with the response:

"Refunds are processed within 5 business days [1] for orders
 returned within the 30-day window [2]."
 [1] returns-policy.md § Processing times (updated 2025-09-14)
 [2] returns-policy.md § Eligibility

Three payoffs:

The user can verify. A cited answer is checkable; an uncited one must be trusted.

Ungrounded generation becomes detectable. A claim with no citation, or a citation that does not support it, is a signal you can measure — which converts the invisible failure above into a monitorable one. You can compute a grounding rate and alert when it falls.

Debugging becomes possible. When an answer is wrong, citations tell you immediately whether the failure was retrieval (the right chunk was never fetched) or generation (it was fetched and misused). Those have completely different fixes, and without citations you cannot tell them apart.

That third point connects to Lesson 4: feedback carries a message_id, so if citations and retrieval scores are logged per message, "do thumbs-down answers correlate with low retrieval scores?" becomes answerable.

Citations turn grounding from an assumption into a measurement — which is the same move the rest of this module makes repeatedly: convert a silent failure into a visible one.

Keeping the index fresh

A knowledge ingestion pipeline updates these embeddings whenever source documents change, keeping the retrieval layer up to date.

Document changes are handled; embedding-model changes are not

Re-embedding on document change is correct and it covers only one of the two ways an index goes stale.

The other is changing the embedding model itself — and it is a full-corpus operation:

Vectors from model A and model B are NOT COMPARABLE.
They are points in different spaces.

Upgrade the embedding model ->
  every chunk must be re-embedded, or search returns nonsense

You cannot migrate incrementally, because a query embedded with the new model searched against a mixed index matches the newly-embedded chunks and effectively ignores the rest — a silent, partial retrieval failure that looks like a quality regression rather than a migration bug.

The safe pattern is the one used for every index rebuild in this module:

1. Build a NEW index with the new model, alongside the old
2. Evaluate retrieval quality on a held-out query set
3. Swap the pointer atomically
4. Keep the old index for rollback

That is that building block's trie publication and that building block's blue-green deployment, applied to a vector index: build beside, swap a pointer, keep the old one for rollback.

An embedding model is a schema for your vector index, and changing it is a full migration — which is why teams change it rarely and evaluate carefully.

Top-k is a three-way trade and the design never picks a number

k — how many chunks to inject — is the pipeline's main tuning knob, and every lesson so far has a stake in it:

k TOO SMALL:  the answer isn't in the context -> retrieval failure ->
              the model falls back to parametric memory
k TOO LARGE:  prompt grows -> TTFT grows (Lesson 3, prefill scales with
              prompt length) -> cost grows (Lesson 6) -> and the relevant
              chunk competes with irrelevant ones for the model's attention

That last effect is real and underappreciated: more context is not monotonically better. Models attend less reliably to material buried in the middle of a long prompt, so padding with marginal chunks can make a retrievable answer harder to use.

The refinement production systems add is a reranker: retrieve a generous candidate set (say 50) with fast approximate search, then rescore with a slower, more accurate cross-encoder and keep the best 3–5.

Vector search:  fast, approximate, high recall     -> get candidates
Reranker:       slow, accurate, high precision     -> pick the best few

Retrieve broadly and rank precisely — it decouples "did we find it at all" from "did we give the model the right few," and it is the standard answer when k is hard to choose.

Next: cost-aware routing and the caching technique the chapter reaches for without naming.