Fetching From Servers You Do Not Own
In one line: every previous chapter designed a system where you controlled both ends. A crawler's throughput is limited by servers belonging to strangers, and that single fact produces most of its interesting constraints.
What it does
A web crawler is a bot that systematically scours the World Wide Web for content, starting from a pool of seed URLs. Crawlers fetch web pages, parse content, and extract URLs for further crawling. This is the foundation of search engines.
The crawler's output feeds downstream stages this chapter deliberately excludes: data cleaning, indexing, relevance scoring (PageRank), URL frontier management, and analytics.
The defining constraint: you are a guest on infrastructure you do not control
Every system so far in this module was bounded by resources you own — your GPUs, your bandwidth, your database. A crawler is bounded by other people's servers, and that changes the nature of the problem.
Three consequences run through the whole chapter:
You must throttle yourself. The non-functional requirements call it "self-throttling to limit crawling per domain." Fetch too fast from one host and you degrade or take down a site that never agreed to serve you. Lesson 9's politeness mechanism — adjusting crawl rate by the domain's time-to-first-byte — is a system that deliberately runs slower than it could.
You must obey rules you did not write. robots.txt is a voluntary protocol with no enforcement. A crawler follows it because not following it is antisocial and gets you blocked, not because anything stops you.
The other side can be hostile. Lesson 9's crawler traps include ones that are "intentional (malicious)" — infinite URL spaces designed to waste your resources. This is the second adversarial system in the module after Uber's fraud detection, and here the adversary is a website.
When your throughput depends on infrastructure you do not own, restraint becomes a design requirement rather than a courtesy. That is the sentence to carry through the chapter.
Why it is a classic interview question
| Benefit | Detail |
|---|---|
| Web page testing | Validating links and HTML structures |
| Web page monitoring | Tracking content or structure updates |
| Site mirroring | Creating mirrors of popular websites |
| Copyright checks | Detecting unauthorized content usage |
| Challenge | Detail |
|---|---|
| Crawler traps | Infinite loops caused by dynamic links or calendar pages |
| Duplicate content | Repeatedly crawling the same pages wastes resources |
| Rate limiting | Fetching too many pages from a single domain overloads servers |
| DNS lookup latency | Frequent DNS lookups slow down the process |
| Scalability | Millions of seed URLs distributed across multiple servers |
Four of the five challenges are about restraint, not capability
Read the list again. Only scalability is about doing more. The other four are about not doing damage or wasting effort:
- Traps waste your resources — stop early.
- Duplicates waste your resources — do not refetch.
- Rate limiting protects their servers — go slower.
- DNS latency wastes time — cache rather than ask again.
That is an unusual balance. Most systems in this module were about maximizing throughput; here the design is largely about knowing when to stop.
It also explains why the components in Lesson 4 look the way they do. A duplicate eliminator, a scheduler enforcing frequency, a DNS cache, and trap detection are all mechanisms for not fetching — and only the HTML fetcher actually fetches.
A crawler is mostly a system for deciding what not to download.
What this chapter gets wrong, and it is worth knowing early
Lesson 3 works through it, but the shape is worth previewing because it is a new variety of error.
The chapter computes that fetching 5 billion pages at 60 ms each takes 9.5 years on one machine, so you need 3,468 servers to finish in a day.
That treats 60 ms as work. It is latency — a fetching thread spends nearly all of it blocked on the network, doing nothing. And the chapter's own non-functional requirements say the system must be "distributed and multithreaded."
1 concurrent fetch per server -> 3,468 servers <- the published figure 100 concurrent fetches -> 35 servers
Meanwhile the chapter separately computes 958 Gb/s of bandwidth and never connects it to the server count — and that is the figure that actually binds. At a 10 Gb/s network interface you need about 96 servers just to move the bytes.
The honest answer is roughly 100 servers, bandwidth-bound. The published 3,468 is latency-bound with a threading model the requirements explicitly rule out.
The system's defining shape is that the output feeds the input. A crawler is a loop, not a pipeline, and almost every hard problem in the chapter comes from that: traps, duplicates, and unbounded growth are all consequences of a process that generates its own work.
Key takeaway
A crawler is the first system in the module bounded by infrastructure you do not own, which makes restraint a design requirement: self-throttling, obeying a voluntary protocol, and defending against websites that are sometimes deliberately hostile. Four of the five stated challenges are about not doing something — which is why most components exist to avoid fetching, and only one actually fetches. And the chapter's server estimate treats network latency as serialized work, producing 3,468 servers where the multithreading its own requirements demand gives ~35 — while the figure that actually binds, 958 Gb/s of bandwidth, is computed and never connected.
Next: the requirements, and where seed URLs come from.