The Workflow

In one line: the seventh step turns a linear pipeline into a cycle, and that is what makes a crawler a fundamentally different shape from every other system in this module.

The seven steps

Step 7 closes the loop, and that is the system's defining shape

Every other design in this module is a pipeline: a request enters, work happens, a response leaves. A crawler is a cycle — its output is its own input.

fetch a page -> extract its links -> those links become work -> fetch them -> ...

Three consequences follow, and none applies to a pipeline.

There is no natural termination. A crawl ends when you decide it ends, not when the work runs out.

Growth is potentially exponential. One page yielding ten new links, each yielding ten, grows without bound. That is why Lesson 7's dedup and Lesson 9's trap detection are not optimizations — they are what keeps the loop from diverging.

Errors compound. A pipeline that mishandles one request affects one request. A crawler that walks into a trap generates work that generates more work. Feedback amplifies mistakes.

A self-feeding loop needs a governor, and in a crawler the governor is everything that prevents fetching: deduplication, trap detection, depth limits, and robots.txt.

That reframes Lesson 4's observation. One component of eight fetches not because the others are support functions, but because in a feedback system, the control mechanisms outnumber the actuators.

Failure, retries, and why the pipeline is staged

A crawler talks to the open internet, so failure is the normal case rather than the exception. The architectural answer is to stage the pipeline and keep no progress in memory.

Each stage is independently retryable because the message is not removed from the queue until the stage acknowledges it. A fetcher that dies mid-download leaves the URL in the frontier; an extractor that dies leaves the HTML already durable in the blob store and the message still queued. Nothing is lost and nothing is redone from scratch.

Three mechanics worth naming, because they are what makes retrying safe at scale:

Visibility timeout. A claimed message is hidden rather than deleted, so a worker that dies silently releases its work when the timeout lapses. Extend the timeout while genuinely still working.

Bounded retries with exponential backoff, then a dead-letter queue. A host that is down should not be retried forever at full rate. Back off, cap the attempts, and move the URL aside for inspection rather than letting it cycle indefinitely.

Jitter. Retries scheduled at exactly the same delay synchronize, so a host recovering from an outage is hit by a thundering herd of crawlers at once — which is both impolite and self-defeating. Add a random offset to every backoff.

Recrawling makes step 7 two different loops

Read step 7 carefully — it describes two things:

The crawler goes back to the first point and repeats until the URL frontier queue is empty. The URLs stored in the scheduler's database have priority and periodicity assigned to them. Enqueuing new URLs into the URL frontier depends on these two factors.

The discovery loop: new URLs found by crawling become new work. This is the expansion.

The refresh loop: URLs already crawled are re-enqueued when due, based on periodicity. This is the maintenance.

They have different characters:

And they compete for the same finite crawl capacity. Every fetch spent revisiting a known page is a fetch not spent discovering a new one.

That is the crawler's fundamental allocation problem and the design does not name it: how much of your capacity goes to breadth versus freshness? A search engine that only discovers has a stale index; one that only refreshes never grows.

When a system has two loops competing for one resource, the split between them is a policy decision that should be explicit.

Step 5 discards on any match, which is too aggressive for recrawling

The duplicate eliminator discards the incoming request in case of a match.

That is right for discovery and wrong for refresh.

If a URL is being recrawled deliberately — because its periodicity says it is due — then finding its checksum in the URL store is expected, not a reason to discard. The whole point of revisiting is that you have seen it before.

Similarly for documents: if a page is recrawled and its content is unchanged, the document checksum matches. Discarding is correct for storage — you do not need a second copy — but the system should still record that the page was checked and found unchanged, because that is exactly the signal Lesson 2's frequency estimation needs.

Naive:    checksum matches -> discard, learn nothing
Better:   checksum matches -> skip storage, UPDATE last-crawled and
                              lengthen the recrawl interval

A negative result is still information. A page that has not changed in five visits should be visited less often, and that inference is only possible if the unchanged result is recorded rather than dropped.

The design's dedup and its recrawl scheduling are both present and never connected. Deduplication in a system that revisits must distinguish "seen before" from "unchanged since."

Client-side load balancing, mentioned in passing

Note: Given the microservices architecture, the design can utilize client-side load balancing.

A single-line callback to that building block's deterministic aperture, and it fits here for the same reason it fit there: many services calling many services, where a centralized balancer on every internal hop would be a bottleneck and an extra network round trip.

Worth noting because the crawler's internal call graph is genuinely busy — the service host calls the DNS resolver, the fetcher, the extractor, and the duplicate eliminator for every page, at whatever rate the fleet is crawling. That is exactly the internal-traffic amplification that motivated moving load balancing into the client.

Next: crawler traps.