Resource Estimation

In one line: this estimate makes a new kind of error — one that will not appear anywhere else in the module — and it computes the correct binding constraint two sections later without noticing.

Assumptions

• Total web pages: 5 billion
• Text content per page: 2,070 KB
• Metadata per page: 500 bytes

Storage

5,000,000,000 x (2,070 KB + 500 B) = 10.35 PB per crawl

2,070 KB of text per page is 20 to 100 times too large — and it is 99.98% of the answer

Two megabytes of text on an average web page. Not the page weight including images, scripts, and fonts — the chapter says "text content."

Real HTML documents average tens of kilobytes. A long article with markup might reach 200 KB; a typical page is well under 100 KB.

And this one number is essentially the entire estimate:

Text:     5B x 2,070 KB = 10.35 PB    <- 99.98%
Metadata: 5B x 500 B    =  2.5 TB     <-  0.02%

Recompute with plausible values:

So the storage figure is plausibly twenty times too high, and everything downstream inherits it — including the bandwidth number, which is derived directly from it.

The habit is the one from that building block, in a new place: find the term that dominates and interrogate it first. Here the metadata line could be wrong by a factor of a hundred and move the total by 2%.

A note on charity: it is possible the original figure was 2,070 bytes (about 2 KB, plausible for extracted plain text with markup stripped), which would give 10.35 TB rather than PB. Either way, the published PB figure does not follow from a defensible per-page size, and it is worth flagging rather than carrying forward.

Traversal time

Assuming an average HTTP traversal time of 60 ms:

5,000,000,000 x 60 ms = 300,000,000 seconds = 3,472 days = 9.5 years

A single instance would take 9.5 years. To complete the task in one day, we need a multi-worker architecture.

Server estimation

One server takes 3,468 days  ->  3,468 servers to finish in one day

Sixty milliseconds is latency, not work — and the requirements already said so

This is the chapter's central error, and it is a variety that has not appeared elsewhere in the module.

The calculation treats 60 ms as occupied time: the server is busy for 60 ms per page, so 5 billion pages take 5 billion × 60 ms of server-time.

But an HTTP fetch is I/O-bound. During those 60 ms the machine issues a request and then waits — for DNS, for the TCP handshake, for the remote server to respond, for bytes to arrive over the network. The CPU is idle for essentially all of it.

Which means one machine can have hundreds of fetches in flight simultaneously, each independently waiting. That is precisely what the non-functional requirement in Lesson 2 demanded: "distributed and multithreaded."

Concurrent fetches per server    Servers needed
              1                      3,468      <- the published figure
             10                        347
            100                         35
            500                          7

At 100 concurrent fetches — modest for an I/O-bound workload — the answer is 35 servers, not 3,468. A factor of a hundred, and it comes entirely from a threading model the chapter's own requirements specify and its estimate ignores.

The general form is worth naming, because it recurs whenever people size systems that talk to networks:

Latency tells you how long one operation takes; it does not tell you how many can run at once. Multiply latency by count and you have computed the time for a strictly serial execution, which is almost never the design.

The correct question is not "how long does one fetch take?" but "how many can I have outstanding, and what limits that number?" — and the answer is usually memory, file descriptors, or bandwidth, not the latency itself.

Bandwidth

10.35 PB / 86,400 s = 120 GB/s = 958 Gb/s total
958 Gb/s / 3,468 servers = 277 Mb/s per server

This is the number that actually binds, and the chapter never connects it

The chapter computes 958 Gb/s, divides it across its 3,468 servers, gets a comfortable 277 Mb/s each, and stops.

Turn the calculation around and it answers the server question properly:

Total bandwidth needed:  958 Gb/s

At  10 Gb/s per server ->  96 servers   just to move the bytes
At  25 Gb/s per server ->  38 servers
At 100 Gb/s per server ->  10 servers

Roughly 100 servers with commodity 10-gigabit networking — and no amount of threading reduces that, because it is a volume constraint rather than a latency one.

So the two sections give different answers to the same question and never meet:

The bandwidth figure is the honest one, because it is the constraint that cannot be engineered away by concurrency. You can always add threads; you cannot exceed your network interface.

And note the interaction: threading reduces the server count until you hit the bandwidth floor, then stops helping. Concurrency converts a latency-bound system into a bandwidth-bound one, and the bandwidth floor is where the real answer lives.

That reframing is the most useful thing in this lesson. It is also worth noting that with the corrected page size from earlier, the bandwidth falls proportionally — at 100 KB per page, 958 Gb/s becomes about 47 Gb/s, and the fleet drops to a handful of machines.

Where the honest numbers land

What the estimate never sizes at all

Three quantities a real crawler plan would need, and none appears:

DNS query rate. Lesson 6 makes DNS caching central. At 5 billion pages a day, even a 90% cache hit rate means 5.8 thousand DNS queries per second — enough to matter, and enough to justify the custom resolver the design specifies.

The checksum stores. Lesson 7's duplicate eliminator holds a checksum per URL and per document. At 5 billion of each and 8 bytes per checksum, that is 80 GB of pure lookup structure, queried on every single fetch — and it must be fast, because it sits on the hot path.

The URL frontier's real size. Lesson 5 sizes it at 1 million URLs. But a crawl that discovers 5 billion pages extracts vastly more links than that — the frontier is a working set, and how you bound it is a design decision the chapter does not make.

An estimate that sizes only the obvious resources will miss the ones on the critical path.

The estimate that fails is the one using theoretical line rate. Real crawlers spend most of their time waiting — on DNS, on slow hosts, on politeness delays — so per-machine throughput lands at a fraction of what the network card could do, and the machine count follows from that fraction.

Interview signal by level

Next: the components.