Crawler Traps and Politeness

In one line: this is the chapter's adversarial section, and its politeness mechanism is the most elegant idea in the design — a throttle driven by a signal from the party being throttled.

Traps

A crawler trap is a URL structure that causes indefinite crawling, exhausting resources.

A calendar is an infinite URL space generated by a finite website

The calendar case is the clearest illustration and worth dwelling on.

A calendar page has a "next month" link. That page has a "next month" link. There is no last page — the URL space is unbounded while the site's actual content is small and finite.

/calendar/2024/01  ->  /calendar/2024/02  ->  ...  ->  /calendar/9999/12  ->  ...

Nothing here is malicious. Nobody built a trap. The infinity is emergent from a perfectly reasonable feature meeting a crawler that follows every link.

Which is why the design classifies traps as "often result from poor website structure" and adds that they can be accidental or intentional (malicious). The accidental ones are far more common.

Two things this exposes about the crawler's model of the world:

"Number of pages" is not a well-defined quantity. Lesson 3 assumed 5 billion pages. But with dynamic URL generation, the number of distinct URLs is effectively infinite while the number of distinct documents is not. The crawler is enumerating a space it cannot bound.

The link graph is not the content graph. A crawler traverses links, but what it wants is content. Those diverge exactly where traps live — many URLs, little content.

When a system enumerates a space defined by other people, that space may be infinite even when their intentions are benign.

Identification

We identify traps by analyzing:

1. URL scheme: detecting patterns like cyclic directories.
2. Page count: flagging domains with an implausible number of pages.

Two detectors, catching structure and volume

They target different signatures and both are needed.

URL pattern detection catches the shape of a trap. A path containing /first/second/first/second/ is self-similar in a way real site structure rarely is. Cheap, local to one URL, and catches cyclic directories immediately.

Page-count detection catches the volume — a domain that has yielded a million pages when comparable sites yield thousands. It requires per-domain state, which Lesson 4 noted no component owns.

The second is the more general detector, because it catches traps whose URLs look perfectly normal. A calendar's URLs are not self-similar; /calendar/2027/03 is a plausible-looking path. Only the count gives it away.

And it connects to Lesson 5's partitioning: because one worker owns a whole hostname, per-domain page counts are local state on that worker — no coordination needed. The same partitioning key that made politeness local makes trap detection local too.

The three defences

robots.txt is voluntary, and the design says so

Note: robots.txt does not prevent malicious traps. Other mechanisms must handle those.

That is an important admission. The robots exclusion protocol is a convention with no enforcement. Nothing prevents a crawler from ignoring it, and nothing prevents a hostile site from writing one that lies.

So it defends against accidental traps — a site politely telling you "the calendar is not worth crawling" — and does nothing against deliberate ones, where the operator wants you stuck.

That leaves the application logic limits doing the real defensive work, and it is the right layering:

robots.txt      -> cooperation with well-behaved sites
depth/count caps -> defence against everything else

The general form: a voluntary protocol is a cooperation mechanism, not a security mechanism. Honour it because not honouring it is antisocial and gets you blocked, but never depend on it for protection.

Worth noting a cost the design does not mention: robots.txt must be fetched per domain and cached, and re-fetched periodically since it changes. That is a component Lesson 4 found missing from the block list — and it is on the critical path, because you cannot fetch anything from a host until you know what that host permits.

Crawl-delay is the directive most crawlers ignore and the one that makes politeness concrete: the host is telling you its preferred minimum gap between requests, and honouring it costs you nothing given how many other hosts you could be fetching meanwhile.

The per-host lock exists for a race condition that only appears in a distributed crawler. Hostname partitioning makes politeness local if exactly one worker owns a host — but during rebalancing, retries, or a redirect that crosses hosts, two workers can end up holding URLs for the same host. A short-lived lock with a TTL matching the crawl delay closes that window, and a failed acquisition simply defers rather than blocks.

And jitter on the deferral matters for the same reason it matters on retries: without it, every deferred URL for a host wakes at the same instant.

Politeness driven by time-to-first-byte is the design's best idea

Politeness: Adjust crawl speed based on the domain's time to first byte (TTFB). Slower servers receive slower crawls to avoid timeouts and overload.

This is genuinely elegant and worth recognizing as a pattern.

The problem: you must not overload servers you do not own, but you have no idea what they can handle. A rate that is trivial for a large site could take down a small one. And you cannot ask.

The insight: the server tells you, through its response time. TTFB is a direct signal of how much load the host is under — and critically, of whether you are the cause.

TTFB rising   ->  the host is struggling  ->  SLOW DOWN
TTFB stable   ->  the host is comfortable ->  can go faster

That makes it a closed-loop controller using the victim's own health as feedback. Three properties fall out:

It is self-calibrating. No per-domain configuration, no guessing at capacity. A small blog and a large news site are both handled correctly because both report their own strain.

It degrades gracefully. If a host slows for reasons unrelated to you, you back off anyway — which is the right behaviour, since it is under load from something.

It is exactly the shape of TCP congestion control, which infers available bandwidth from loss and delay rather than being told. When you cannot measure a remote resource's capacity, infer it from the resource's own response.

The one caveat worth adding: a closed loop can oscillate, the same failure that building block identified in traffic routing. Speed up because TTFB improved, overload the host, back off, repeat. Real implementations damp this — additive increase, multiplicative decrease, again like TCP.

The IP-blocking question the design poses

Consider a real-time news aggregation platform where frequent recrawls are crucial. What steps would you take if the IP faces blocking issues during frequent crawls?

Worth answering, because the first instinct is usually wrong.

The wrong answer is evasion — rotating IPs, proxies, spoofing user agents. It works briefly, escalates the conflict, and is what makes crawlers unwelcome.

The right answers, in order:

Ask why you were blocked. Almost always: crawling too fast, ignoring robots.txt, or not identifying yourself. All three are fixable by behaving better, and the politeness controller above is the mechanism.

Identify yourself honestly. A named user agent with a contact URL lets an operator tell you to slow down instead of blocking you.

Negotiate. Large publishers often offer feeds, sitemaps, or APIs that are cheaper for both sides than crawling. For a news aggregator specifically, RSS and sitemaps solve the freshness problem far better than frequent recrawling — they tell you what changed instead of making you check.

Then back off and retry with exponential delay, treating the block as a strong politeness signal rather than an obstacle.

When an external party blocks you, the default assumption should be that you earned it. That answer is both more honest and more likely to work than an evasion strategy.

Next: extensibility and distributing work.