Ingestion: Parsing the Documents
In one line: retrieval quality is capped by what the parser extracted, and a table flattened into unreadable prose at ingestion is a wrong answer six months later that nobody traces back here.
The format determines the difficulty
| Source | Difficulty | What goes wrong |
|---|---|---|
| Markdown, plain text | Trivial | Nothing — structure is explicit |
| HTML | Easy | Navigation, ads and boilerplate become content |
| Word documents | Moderate | Tracked changes, comments, embedded objects |
| Slides | Moderate | Text is positional; reading order is a guess |
| Hard | It describes marks on a page, not a document | |
| Scanned images | Hardest | OCR errors compound into everything downstream |
PDF deserves its reputation and it is worth being able to say why. A PDF is a description of where glyphs sit on a page — it has no reliable notion of paragraph, heading, column or table. Everything a parser gives you back is inferred from position, and every inference can be wrong.
The four PDF failures
Reading order. A two-column page can extract as interleaved lines from both columns, producing text that is locally grammatical and globally nonsense. It embeds fine and retrieves fine and means nothing.
Tables. The most consequential. A table is a grid of relationships, and flattened to a line of text it becomes an unreadable sequence of numbers with no association to their headers. A chunk containing a mangled pricing table is worse than one containing nothing, because it will be retrieved for pricing questions.
Headers, footers and page furniture. Repeated on every page, they become the most frequent text in the corpus, and they dilute embeddings while matching nothing anyone wants.
Figures and their captions. The information is in the image; the caption alone is a fragment. Without a vision model the content is simply absent, and the caption retrieves as if it were present.
What to preserve
The parser's job is not to produce text. It is to produce text plus the structure that makes the text interpretable.
Headings and hierarchy. Which section a passage belongs to, and the path down to it. This is what later lets a chunk carry its own context.
Tables as tables. Extract to a structured form — Markdown or HTML — so the row and column relationships survive. A model can read a Markdown table; it cannot read a flattened one.
Source location. Document id, page number, section. This is what makes a citation possible, and citation is what makes the system trustworthy.
Document metadata. Author, dates, type, product area, language, and crucially who is allowed to see it — which the permissions lesson depends on entirely.
Capture metadata at ingestion or lose it forever
The point worth making loudest. Metadata that exists in the source document and is not captured during ingestion is generally unrecoverable — you would have to re-parse the whole corpus to get it.
So the guidance is to over-capture. Storage for metadata is trivially cheap next to the cost of a re-ingest, and the fields you will wish you had are predictable: timestamps for freshness filtering and recency ranking, access-control identifiers, document type for routing, and language for multilingual filtering.
Keeping the raw source text alongside falls in the same category, and for the same reason the retrieval chapter gave: it is what makes a change of embedding model survivable.
Deciding how much to invest
Parsing quality is a spectrum with a real cost curve, and picking a point on it deliberately is the answer.
| Approach | Cost | When |
|---|---|---|
| Basic text extraction | Near zero | Clean, structured sources |
| Layout-aware parsing | Moderate | Mixed documents, some PDFs |
| Vision-model parsing | High — a model call per page | Complex layouts, scans, tables that matter |
| Human-in-the-loop correction | Highest | A small, high-value, high-stakes corpus |
The decision is driven by consequence rather than volume. A hundred contracts where a misread clause is a legal exposure justifies vision parsing and spot-checking; a million forum posts do not.
Key takeaway
A PDF describes glyph positions rather than a document, so reading order, tables, page furniture and figures are all inferred and all fail silently — and nothing downstream can tell. Preserve structure rather than just text: heading hierarchy, tables in a form a model can read, and source location so citation is possible. Over-capture metadata at ingestion, because what you skip costs a full re-parse to recover. And choose parsing investment by the consequence of an error, not the size of the corpus.
Next: the chunking decision, and why most retrieval failures start there.