The URL Frontier

In one line: the partitioning choice here is the design's best decision, because it converts a distributed coordination problem into a local one.

How big is it?

Assumption: at any point we have roughly one million URLs in the URL frontier.

1,000,000 URLs x 2,048 bytes = 2.048 GB

2.048 GB is a reasonable amount of space for a queue, indicating we might not need a distributed mechanism for the URL frontier.

2,048 bytes per URL is generous, and the conclusion holds anyway

A URL averages around 70 characters. Even with metadata — priority, recrawl frequency, last-crawled timestamp, discovery source — a frontier entry is a few hundred bytes at most.

At 2,048 B (published):  2.048 GB
At   200 B (realistic):  200 MB

Either way the conclusion is right and worth stating plainly: the frontier fits in memory on one machine. That is unusual for a system processing billions of pages, and it is because the frontier is a working set — only the URLs due for crawling right now — not the full catalogue, which Lesson 4 established lives in the database.

So centralization is technically viable. The design distributes anyway, and the reasons are not about size.

Why distribute it regardless

A centralized queue has limited read/write bandwidth and is a single point of failure. Therefore, having a sub-queue for each worker will be the best approach.

Three reasons, and only one is capacity

The third is the most interesting and the design states it well: "having independent queues can further optimize the crawling process, especially in the case of high priority and more frequent crawls like news websites that need more than regular workers."

That is workload isolation. News sites need crawling many times a day; ordinary pages every two weeks. Mixing them in one queue means the high-frequency work competes with the bulk work for the same dequeue capacity.

Separate queues let you dedicate workers to the fast lane, and it is the same argument as priority classes in a task scheduler.

Notice also the honest tension the design raises: "having a single queue is beneficial for the deduplication of redundant links." One queue makes "have I already got this URL?" a local lookup. Split it, and dedup must happen somewhere else — which is exactly why Lesson 7's duplicate eliminator is a separate component with its own store.

Partitioning a queue moves deduplication out of it.

The partitioning rule

Distributing the frontier involves computing a hash of each URL's hostname and assigning URLs to worker nodes based on that hash. Each worker maintains its own sub-queue.

This fulfills two requirements:

• Workers don't individually connect to more than one host web server at a time.
• Workers don't overburden host servers with concurrent requests, because we use FIFO sub-queues.

Hashing on hostname rather than full URL is the key move. It guarantees every URL for a host lands on one worker, so rate limiting that host is a local decision rather than a distributed one. The cost is skew — a few enormous hosts get disproportionate queues.

Hashing on hostname makes politeness a local property — this is the key move

Lesson 2 established that per-host rate limiting is the central constraint: you must not overload servers you do not own.

Enforcing that across a distributed crawler could be genuinely hard. If any worker can fetch from any host, then "no more than N requests per second to example.com" is a global invariant across thousands of machines — requiring shared counters, coordination, and consensus on a hot path.

Hashing on hostname eliminates the problem entirely:

All URLs for example.com  ->  hash("example.com")  ->  ALWAYS worker 7

Now only one worker ever talks to example.com. So the rate limit is a local decision made by that worker with no coordination whatsoever — it just paces its own FIFO queue.

A global invariant becomes a local one by partitioning on the dimension the invariant is about. That is the general principle, and it is the same reason that building block partitioned by geography and that building block partitions by key: choose the partition key so that the thing you must coordinate lives inside one partition.

Two further benefits fall out for free:

Connection reuse. One worker talking repeatedly to one host can keep a persistent HTTP connection open, avoiding repeated TCP and TLS handshakes. That is worth more than it sounds when handshakes are a large fraction of a 60 ms fetch.

Cached DNS and robots.txt. Both are per-host, and both are now needed by only one worker — so a single cache entry serves every request to that host.

What hostname partitioning costs: skew

The web's host distribution is extremely uneven. A handful of domains have hundreds of millions of pages; most have a handful.

So hashing on hostname produces workers with wildly different queue depths:

Worker holding a giant domain  ->  hundreds of millions of URLs
Worker holding small sites     ->  thousands

That is the same power-law problem every chapter in this module has met, and the design does not address it. Three partial answers worth knowing:

Politeness caps the giant anyway. A worker owning a huge domain cannot crawl it quickly regardless, because it must rate-limit itself. So the imbalance in queue depth does not translate directly into imbalance in work rate — the big worker is idle-waiting, not overloaded.

Split very large domains by subdomain or path prefix. At the cost of needing coordination for politeness across those splits, which is exactly what the scheme was avoiding.

Consistent hashing with virtual nodes, which Lesson 11's evaluation mentions — it makes adding and removing workers cheap but does not fix skew on its own.

Partitioning on a naturally skewed key gives you locality and inherits the skew. Worth naming as the cost of the decision rather than pretending it does not exist.

FIFO within a sub-queue, priority across the frontier

Notice the two ordering disciplines operating at different levels:

Across the frontier: URLs enter based on priority and recrawl frequency — a priority queue.

Within a worker's sub-queue: strict FIFO, which "*

That combination is deliberate. Priority decides which URLs are due; FIFO decides the order one worker hits a host, and FIFO is what makes pacing predictable — you cannot rate-limit a host if the queue keeps reordering.

The alternative the design offers is worth noting: "separate queues for different priorities... dequeue based on the priorities assigned to them," which "just requires the URL placement in the respective queue and doesn't need scripts to schedule based on extra parameters."

That is the classic trade: encode priority in the data (one queue, a priority field) or in the structure (several queues, one per class). Structure is simpler to operate and coarser; data is finer-grained and needs scheduling logic. The design's closing line is right — "it all depends on the scale."

Next: DNS and fetching.