Chunking
In one line: a chunk is the unit you retrieve, so its size decides precision and its isolation decides whether it means anything at all.
Why chunk
Two hard reasons and one soft one.
The embedding model has a maximum input length. And embedding a long passage into one fixed-length vector averages away its specifics — a 4,000-token chunk becomes a vector about the general topic and stops matching precise questions.
The soft reason is cost: retrieving whole documents fills the context window with mostly irrelevant text you pay for on every request.
The size trade-off
There is no universal answer, and roughly 500 tokens with some overlap is a reasonable default to start measuring from rather than a recommendation to settle on.
Overlap — repeating some tokens between adjacent chunks — exists so that a passage split across a boundary still appears intact in one of them. It costs storage proportional to the overlap fraction and it is cheap insurance. Ten to twenty percent is typical.
The right size depends on the question shape, and saying so is better than quoting a number:
| Question shape | Chunk size |
|---|---|
| Precise factual lookup — a value, a date, a setting | Smaller |
| Explanatory — how something works, why a decision was made | Larger |
| Procedural — a sequence of steps | Whole procedure, never split mid-sequence |
That last row generalises into the most useful rule available: chunk on structure, not on length. A section, a procedure, a table, a function — these are units an author already decided were coherent. Splitting every 500 tokens regardless cuts through the middle of them.
The isolation problem
The deeper failure, and the one that explains most retrieval misses.
A chunk is retrieved alone, so it must be interpretable alone. Text written as part of a document rarely is.
"It increased by 12% in that period, largely due to the factors described above."
Retrievable, and useless. What increased? Which period? Which factors? The chunk embeds as a vector about increases and percentages, matches almost nothing anyone actually asks, and if retrieved contributes nothing.
The general form: anaphora — pronouns, references and implicit subjects — breaks when a chunk leaves its document. Anaphora — "it", "this", "the above" — is everywhere in real writing, and it is exactly what does not survive the split.
Putting the context back
Three approaches, in increasing sophistication and cost.
Prepend the structural path
The cheapest and the one everyone should do. Add the document title and heading hierarchy to each chunk before embedding:
Acme API Guide > Authentication > Token Refresh It increased by 12% in that period...
Now the chunk embeds with its subject attached, and it matches queries about Acme authentication. This costs nothing beyond having captured the hierarchy at ingestion — which is why the previous lesson insisted on it.
Contextual retrieval
Use a model to write a short situating summary for each chunk and prepend it before embedding.
It works well and its cost is honest: one model call per chunk at ingestion. On a million-chunk corpus that is a real spend, though a one-off, and it is dramatically cheaper with prompt caching since the document is the shared prefix across all its chunks.
Late chunking
Embed the whole document at token level first, then segment the token embeddings into chunks and pool each one.
Because every token's embedding was computed with the full document visible, references and pronouns are resolved before the split ever happens. It improves retrieval accuracy by roughly 10–12% on documents with anaphoric references — precisely the failure above — and it is more compute-efficient than contextual retrieval because it needs no per-chunk model call.
The trade is that it requires an embedding model supporting long inputs and token-level output, and it tends to give up some relevance and completeness relative to contextual retrieval.
| Prepend headings | Contextual retrieval | Late chunking | |
|---|---|---|---|
| Cost | Free | One model call per chunk | One long embedding pass per document |
| Fixes | Missing subject and topic | Missing subject and situating detail | Pronouns and cross-references |
| Needs | Heading hierarchy from parsing | A model and a budget | A long-context, token-output embedding model |
| Do it | Always | When quality justifies the ingest cost | When the model supports it |
The first column is not a lesser option — it is the one that should be in every system, and the other two are additions to it.
What retrieval and generation see can differ
A useful decoupling that resolves the size trade-off rather than balancing it.
Retrieve on small, precise chunks so matching is sharp. Then, before generation, expand each hit to its surrounding context — the neighbouring chunks, or the whole parent section — and pass that to the model.
This gets precision at retrieval and completeness at generation, at the cost of more tokens in the prompt. It is one of the highest-return structural changes available and it needs no new model.
Key takeaway
Chunk on structure rather than length, because sections and procedures are units an author already decided were coherent. Size trades precision against self-containedness with no universal answer — but the deeper problem is isolation: a chunk full of pronouns and implicit subjects is retrievable and meaningless. Always prepend the heading path; add contextual retrieval when quality justifies a model call per chunk, or late chunking where the model supports it, worth 10–12% on documents with anaphoric references. And decouple retrieval size from generation size by expanding hits to their parent section.
Next: the query, which is not shaped like the documents.