Building Blocks and Components

In one line: Lesson 1 observed that a crawler is mostly a system for deciding what not to download. The component list is where that becomes visible.

The blocks

One component fetches; five exist to avoid fetching

Sort the eight by what they contribute:

One out of eight does the thing the system is named for. The rest decide what is worth doing, or remember what has already been done.

That ratio is not an accident. Lesson 3 established the system is bandwidth-bound, and Lesson 1 that it must be polite to hosts it does not own. Under both constraints, the highest-leverage work is not issuing a request:

• A URL you deduplicate costs nothing to fetch.
• A DNS answer you cache costs nothing to resolve.
• A trap you detect saves unbounded fetching.

Compare the streaming and social chapters, where nearly every component was on the delivery path. In a system limited by an external resource, the components that matter most are the ones that reduce demand on it.

The scheduler is two things, and separating them matters

The design decomposes it explicitly:

• Priority queue (URL frontier): hosts URLs ready for crawling, based on priority and update frequency.
• Relational database: stores all URLs along with those two parameters, populated by user-added URLs (seeds and runtime additions) and crawler-extracted URLs.

That is a working set and a catalogue, and they have opposite properties:

The database is the authority on what exists and when it should next be crawled. The queue is a materialized slice of it — the URLs whose time has come.

That is the same materialized-view shape as the newsfeed chapter's precomputed feeds: an expensive authoritative store, plus a small hot subset prepared in advance so the consuming path is fast.

And it is what makes recrawling work. Lesson 2 established this is a continuous system; the database holds every URL with its frequency, and the scheduler repeatedly promotes due URLs into the frontier. Without the database half, a crawler could only ever go forward.

The 'cache' block is ambiguous, and the design uses at least three

The block list says "cache: stores fetched documents for quick access." But the components described use caching for three different things:

A DNS cache (Lesson 6) — hostname to IP, avoiding repeated resolution. This is the one with the clearest justification, since DNS latency is a named challenge.

The Document Input Stream (Lesson 7) — the design names Redis, holding a fetched document while the extractor and duplicate eliminator both work on it. That is not a cache at all; it is a short-lived handoff buffer between pipeline stages.

A content cache — implied by the block description, though nothing in the workflow reads a previously-fetched document back.

The DIS is the interesting one and the description undersells it. It exists because a fetched page is consumed by two components — the extractor pulls URLs and content, the duplicate eliminator checksums it — and passing megabytes between services is worse than parking it once and passing a reference.

A buffer shared between pipeline stages is not a cache; it is how you avoid copying. Same instinct as every chapter's "store references, not values."

What the component list is missing

Two things the design needs and does not name.

A robots.txt fetcher and cache. Lesson 9 makes obeying robots.txt central, which means fetching a separate file per domain and honouring it before every request to that host. That file must itself be cached and periodically refreshed — a component with its own lifecycle, absent from the list.

A trap detector. Lesson 9 describes identifying traps by URL pattern and page count, which requires per-domain state — how many pages fetched, what URL shapes seen. Nothing in the component list owns that state.

Both are mechanisms for not fetching, which by the argument above makes them among the higher-leverage components in the system. The components a design omits are often the ones that do the restraining.

Next: the URL frontier, and why it is partitioned by hostname.