DNS and Fetching

In one line: DNS is listed as one of five named challenges, which is unusual — and the traversal-order decision here contradicts the design's own partitioning scheme.

The DNS resolver

DNS resolver: Maps hostnames to IP addresses. To reduce latency, we use a custom DNS resolver that caches frequently used IPs.

DNS resolution: the worker sends the URL for resolution. The resolver checks the cache and returns the IP if found. Otherwise it determines the IP, returns it, and stores the result in the cache.

Why DNS is a named challenge here and nowhere else in the module

No other chapter lists DNS as a design problem. Here it is one of five, and the reason is structural.

In a normal service, DNS resolution happens once per client session and is cached by the operating system, the browser, and the resolver chain. It is invisible.

A crawler resolves hostnames constantly, because it is contacting a continuous stream of hosts it has often never seen. And a cold DNS lookup is slow — tens to hundreds of milliseconds, involving a recursive walk through the DNS hierarchy.

Put that against Lesson 3's 60 ms fetch budget:

Cold DNS lookup:  50-200 ms
Page fetch:            60 ms

Resolution can cost more than the fetch it enables. That is why a synchronous, uncached resolver would dominate crawl time — and why the design builds a custom one rather than using the system resolver.

Three things a custom resolver buys:

Caching under your control. System resolvers honour TTLs, which for a crawler are often shorter than useful. You may reasonably cache longer, accepting occasional staleness.

Asynchrony. The standard getaddrinfo call is blocking — a thread issuing it stops. Lesson 3's whole argument is that fetching must be concurrent, and a blocking resolver serializes it. A custom resolver can issue many lookups in flight.

Prefetching. You know which hosts are in the frontier, so you can resolve them before a worker needs them.

When a supporting operation costs as much as the operation it supports, it stops being infrastructure and becomes a design problem.

Hostname partitioning makes the DNS cache dramatically more effective

Worth connecting to Lesson 5. Because URLs are partitioned by hostname hash, all requests for example.com go to one worker.

Which means that worker resolves example.com once and reuses it for every subsequent page on that host — and no other worker needs the entry at all.

WITHOUT hostname partitioning:  every worker may need every host -> N copies, N cold lookups
WITH hostname partitioning:     one worker, one entry, one lookup

The same holds for robots.txt and for persistent HTTP connections. One partitioning decision improves DNS caching, robots.txt caching, connection reuse, and politeness simultaneously — which is a strong sign it is the right key.

The HTML fetcher

HTML fetcher: Initiates communication with the host server to download content. While primarily focused on HTTP, it is extendable to other protocols.

Extensibility here is a real requirement, not boilerplate

Lesson 2 listed extensibility as non-functional: "support new protocols (beyond HTTP) and file formats via modular extensions."

Lesson 10 covers the mechanism. What is worth noting now is why a crawler needs it more than most systems: the crawler does not choose its inputs. It encounters whatever the web contains — FTP links, unusual MIME types, formats that did not exist when the crawler was written.

A system that consumes the open web must be extensible, because its input space is defined by other people. Contrast every other chapter, where the system defined its own input formats.

DFS or BFS?

Can we use DFS instead of BFS?

Yes. DFS allows the crawler to stay within the same website for longer, making better use of persistent connections and reducing repeated reconnections.

However, because DFS explores one path deeply before backtracking, it is more susceptible to crawler traps, such as infinite calendar pages, which can delay the discovery of other pages.

In this case, we choose DFS because reducing reconnections is our primary goal.

The stated justification is already delivered by hostname partitioning

The argument for DFS is that it keeps the crawler on one host longer, enabling connection reuse.

But Lesson 5's partitioning already guarantees that. All URLs for a host go to one worker's FIFO sub-queue, so that worker is naturally hitting the same host repeatedly regardless of whether the global traversal is depth-first or breadth-first.

Claimed DFS benefit:  stay on one host -> reuse connections
Already provided by:  hash(hostname) -> one worker owns the host

So the design pays DFS's cost — the design's own admission that it is "more susceptible to crawler traps" — for a benefit it already has.

And the cost is not small. Lesson 9 devotes a whole section to traps, including infinite URL spaces. DFS is exactly the traversal that walks into an infinite branch and never comes back out, because it commits to depth before breadth.

There is a second argument for BFS the chapter does not make, and it is the stronger one: link depth correlates with page importance. Pages a few hops from a well-connected seed tend to be more significant than pages twenty hops down a path. BFS therefore discovers high-value pages first, which matters enormously for a crawl that will never finish — and no crawl of the open web ever finishes.

When a crawl cannot complete, the order in which you discover things is the whole product. BFS optimizes for that; DFS does not.

The honest answer: BFS globally, with per-host locality provided by partitioning — which gives you both properties and neither cost.

The traversal question is partly a false choice

There is a further subtlety worth raising. Once the frontier is a priority queue ordered by priority and recrawl frequency (Lesson 5), the crawler is not doing DFS or BFS in the classical sense.

Classical traversal order is a property of a stack or a queue. Here the ordering discipline is priority, which is neither:

DFS  -> stack      (last discovered, first crawled)
BFS  -> queue      (first discovered, first crawled)
Here -> priority   (most important or most due, first crawled)

So the real question is not depth versus breadth — it is what determines priority. And the design already answers it: recrawl frequency and page importance.

Once you have a priority queue, traversal order is an emergent property of your priority function, not a choice between two algorithms. Saying that is a stronger answer than picking a side.

DNS is the bottleneck people forget. Classic crawler measurements put resolution at a large fraction of total elapsed time, because it is a blocking network round trip before any fetching can begin. A per-fetcher DNS cache plus several resolvers in rotation is the standard answer — the second one matters because a single provider will rate-limit you.

Next: extraction and deduplication.