Extensibility and Evaluation
In one line: the evaluation's performance row says "add more workers" — which is exactly the reasoning Lesson 3 showed produces a 100× overestimate.
Extensibility
Shortcoming: the design supports only HTTP and text extraction.
- HTML fetcher: add modules for other protocols (e.g. FTP). The crawler invokes the correct module based on the URL scheme.
- Extractor: add modules to process non-text media (images, videos) from the DIS, stored in the blob store alongside text.
Dispatch on the URL scheme and the MIME type — the extension points are where the input declares itself
Both extension points key off something the input tells you:
FETCHER: dispatch on URL SCHEME -> http:// https:// ftp:// EXTRACTOR: dispatch on MIME TYPE -> text/html image/jpeg video/mp4
That is the right structure, and the reason is Lesson 6's observation: a crawler does not choose its inputs. It encounters whatever the open web contains, including formats that did not exist when the crawler was written.
So the extension points must sit exactly where the input declares its own type — the scheme in the URL, the Content-Type in the response. Anywhere else and you would have to guess.
When your input space is defined by other people, put your extension points where the input identifies itself. A plugin keyed on a self-describing field is the only kind that survives inputs you did not anticipate.
Two extension points worth naming rather than leaving implicit.
A HEAD request before GET lets you check the MIME type and size before spending bandwidth. Skipping a 200 MB video or a binary you cannot parse costs one cheap round trip and saves the whole download.
JavaScript-rendered pages are invisible to an HTML parser — the links and text simply are not in the markup. Handling them means an extra stage running a headless browser to execute the page before extraction. It is an order of magnitude more expensive per page, which is exactly why it belongs as a separate stage applied selectively rather than as the default path.
Distributing work
| Approach | Detail |
|---|---|
| Domain-level assignment | Assign an entire domain to a worker by hashing the hostname. Prevents redundant crawling and supports reverse URL indexing |
| Range division | Assign ranges of URLs to workers |
| Per-URL crawling | Workers take individual URLs. Requires coordination to avoid collisions |
Only the first preserves politeness, and the design notes the cost of the third
These are presented as three options. They are not equivalent — only one is compatible with the politeness requirement that runs through the whole chapter.
| Domain-level | Range division | Per-URL | |
|---|---|---|---|
| Per-host rate limiting | Local, free | Depends on the range key | Requires global coordination |
| Connection reuse | Yes | Sometimes | No |
| DNS cache hit rate | Excellent | Moderate | Poor |
| Load balance | Skewed (Lesson 5) | Better | Best |
| Coordination | None | Little | "Requires coordination to avoid collisions" |
Per-URL assignment gives the best load distribution and destroys everything else. If any worker can take any URL, then every host may be contacted by many workers at once — and politeness becomes a distributed invariant requiring shared counters on the hot path. The design's own note, "requires coordination to avoid collisions," understates it: the collisions are not just duplicate work, they are politeness violations.
So the apparent trade — better balance versus more coordination — is really better balance versus violating the requirement. Lesson 5's domain-level scheme is the only viable choice, and its skew is a cost you accept rather than a parameter you tune.
When options differ in whether they satisfy a hard requirement, they are not alternatives.
Requirements compliance
| Requirement | Techniques |
|---|---|
| Scalability | Add or remove servers on demand · consistent hashing for server addition and removal · regular S3 backups for fault tolerance |
| Extensibility | New protocol modules in the HTML fetcher · new MIME schemes in the extractor |
| Consistency | Checksums of URLs and documents compared in their respective stores |
| Performance | More workers · blob stores · high priority to robots.txt · self-throttling per domain |
| Scheduling | Pre-defined recrawl frequency, or separate queues per priority class |
'Increase the number of workers' repeats the chapter's central error
The performance row leads with: "URLs crawled per second: we can improve this factor by adding new workers to the system."
Lesson 3 showed why that reasoning is what produced 3,468 servers. The claim is only true up to a point, and the point arrives early:
Add workers -> more concurrent fetches -> higher throughput
-> UNTIL you saturate bandwidth
At Lesson 3's figures the ceiling is 958 Gb/s — roughly 96 machines with 10-gigabit interfaces. Beyond that, additional workers add nothing, because the bytes cannot move any faster.
And there is a second ceiling the row does not mention, which is more interesting: politeness. Lesson 9 established that you deliberately throttle each host. So a crawler's throughput is bounded by
(number of hosts you can talk to in parallel) x (polite rate per host)
Adding workers past that point means workers waiting on their rate limits, not crawling faster. The binding constraint eventually becomes the number of distinct hosts, not the number of machines.
That is worth saying because it is the crawler-specific version of the general lesson: "add more workers" is true until it isn't, and knowing which resource saturates first is the whole answer. Here, in order: bandwidth, then host diversity.
The consistency row is right, and it is right about an unusual meaning
Lesson 2 flagged that "consistency" here means deduplication, not replica agreement. The evaluation confirms it: "to avoid data inconsistency and crawl duplication, our system computes the checksums of URLs and documents."
Which is coherent — and it means the evaluation is honest about what it is claiming. Compare four earlier chapters in this module that claimed replica consistency they did not provide.
The row also adds a genuine fault-tolerance mechanism: "all servers can checkpoint their states to a backup service." For a crawler that is meaningful, because worker state — which URLs are in flight, which hosts are rate-limited — is otherwise lost on a crash and must be rediscovered.
A system whose work is a self-feeding loop must checkpoint, because losing in-flight state means re-deriving it from the loop.
What the evaluation never addresses
Four gaps.
No coverage metric. The evaluation measures URLs crawled per second — a throughput number. But Lesson 2 established that seed quality bounds what fraction of the web you can reach, and Lesson 8 that discovery and refresh compete for capacity. Neither has any measure here. A crawler optimized for pages per second and not for coverage or freshness is optimizing the wrong thing.
No near-duplicate handling. Lesson 7 — exact checksums miss the duplication that dominates the web.
No trap-state ownership. Lesson 9's page-count detection needs per-domain counters, and no component holds them.
No robots.txt component. Named as central in Lesson 9, absent from every list and diagram, and it is on the critical path since you cannot fetch from a host before knowing what it permits.
The first is the most consequential. The evaluation measures the rate of an activity rather than the quality of its result.
Key takeaway
Extension points sit where the input declares its own type — the URL scheme and the MIME type — which is the only structure that survives an input space defined by other people. Of the three work-distribution options, only domain-level assignment preserves politeness; per-URL assignment turns rate limiting into a distributed invariant, so these are not really alternatives. The performance row's "add more workers" repeats the chapter's central error: throughput saturates first at bandwidth (~96 machines) and then at host diversity, since each host is deliberately throttled. The consistency row is honest about meaning deduplication, and correctly adds checkpointing — necessary because a self-feeding loop loses in-flight state. And the evaluation measures pages per second rather than coverage or freshness, which is measuring the activity rather than the result.
Next: the whole design under interview conditions.