Requirements and Seed URLs

In one line: the seed-URL discussion looks like a preliminary. It is actually a statement about graph reachability, and it determines the ceiling on everything the crawler can ever discover.

Functional requirements

Seed quality is a reachability constraint, and the design states it exactly right

When the WWW is modeled as a graph where URLs connect one node to another, the goal is to discover as much of the graph as possible. Using poor seed URLs can limit the crawler to discovering only a small portion of the web graph.

That is precise and it is the most important sentence in the requirements.

A crawler only ever reaches pages linked from something it already has. So the set of pages you can ever discover is the transitive closure of your seeds — everything reachable by following links from where you started.

Which means:

The web is not strongly connected. There are regions with no inbound links from your seeds, and no amount of crawling time reaches them. Seed choice is not an optimization; it is a bound.

Poor seeds cannot be fixed by more workers. Adding servers explores your reachable region faster; it does not enlarge it. This is a coverage problem, not a throughput problem, and they respond to entirely different interventions.

Coverage compounds. A seed that reaches a well-connected hub yields exponentially more than one reaching an isolated page, because each hop multiplies.

The two sources the chapter gives are worth noting for their difference. "Manually create them" is curated and small. "Scan IP addresses for the presence of web servers" is a genuinely different technique — it bypasses the link graph entirely, finding hosts with no inbound links at all. That is the only way to reach the disconnected regions, and it is the reason the second method exists.

When your discovery mechanism is traversal, your starting set determines your ceiling.

Scheduling is a functional requirement, and that makes this a recurring system

"Regularly schedule crawling to update records."

Easy to skim past, and it changes what the system is. A crawler is not a batch job that finishes — it is a continuous process that revisits.

Because the web changes underneath you. A page crawled last month may be gone, moved, or rewritten. So the crawler's real job is maintaining a current snapshot, not producing one.

That produces a scheduling problem with no obvious answer: how often should you revisit a given page? Lesson 11 covers the design's answer — per-URL frequency, with news sites crawled multiple times daily and ordinary pages every two weeks — and the underlying principle is worth naming:

Revisit frequency should track change frequency, not importance. A page that never changes wastes every refetch regardless of how popular it is; a page that changes hourly is stale within an hour no matter how obscure.

In practice you cannot know a page's change rate without observing it, so real crawlers estimate it from history and adjust — which makes recrawl scheduling a feedback loop, like that building block's traffic estimation.

Non-functional requirements

'Distributed and multithreaded' is the requirement Lesson 3's estimate ignores

Note the word multithreaded in the scalability requirement. It is there for a specific reason: fetching is I/O-bound, so a single machine can have hundreds of fetches in flight at once, each mostly waiting.

Then Lesson 3 computes servers by assuming one fetch at a time per machine, arriving at 3,468. Those two statements cannot both be right.

The requirement is correct and the estimate contradicts it. Worth flagging here because it is the chapter's central error and it originates in this sentence being written and then not applied.

When a requirement specifies concurrency, the capacity estimate has to model it.

'Performance' here means going slower, which is unusual

Read the performance requirement carefully:

"Use self-throttling to limit crawling per domain (by time or count) to avoid overloading hosts and optimize throughput."

In every other chapter, performance meant faster. Here it means deliberately slower on any given host.

The two halves are not contradictory, and the reconciliation is the interesting part. Throttling per domain while crawling many domains in parallel means:

Per host:    slow, polite, a few requests per second
In aggregate: fast, thousands of hosts at once

Total throughput comes from breadth, not depth. You get billions of pages by talking to a million hosts gently, not one host aggressively.

And that is exactly why Lesson 5 partitions the URL frontier by hostname — it makes per-host rate limiting a local property of one worker's queue rather than a distributed coordination problem.

A per-resource limit combined with massive parallelism across resources is how you get high aggregate throughput without concentrated load. The same shape as that building block's sharded counters or the rate limiter chapter's per-key windows.

Consistency here means deduplication, not replica agreement

"Ensure data consistency across all crawling workers."

That sounds like the distributed-systems sense — replicas agreeing on a value. It is not. Lesson 11's evaluation makes the intent clear: consistency is achieved "by computing checksums of URLs and documents and comparing them" — that is deduplication.

The actual requirement is: two workers must not independently crawl and store the same thing. Which is a coordination problem, but about work assignment rather than data agreement.

Worth separating, because the mechanisms differ entirely. Replica agreement needs consensus or quorums. Avoiding duplicate work needs partitioning (Lesson 5's hostname hash) plus a shared record of what has been seen (Lesson 7's checksum stores).

When a requirement says "consistency," check whether it means agreement or non-duplication. They are different problems with different solutions.

The central tension: politeness caps per-host throughput, and efficiency demands aggregate throughput. The resolution is that they operate at different granularities — you go slowly at each host and fast across millions of hosts at once.

Next: the estimation, and the two figures that disagree about what limits the system.