Multi-Tier Architecture and the Long Tail
In one line: a flat CDN sends every cache miss to the origin — and with thousands of edges, that is thousands of identical requests for the same object. The hierarchy is what stops the origin from being crushed by its own cache misses.
The hierarchy
Edge proxy servers operate within a hierarchy of parent and peer nodes. They fetch content from upstream parents, which in turn obtain it from the origin. Content propagates from origin to proxies along multiple paths in the hierarchy.
The tree structure lets the system scale for increasing users by adding more server nodes to the tree, and it reduces the burden on the origin server for data distribution. A CDN typically has one or two tiers of proxy servers.
Whenever a new proxy server enters the tree, it requests the control core, which maintains information on all proxy servers in the CDN and provides initial content with the configuration data.
The long tail
Research shows that much content has a long-tail distribution: a handful of items are very popular, followed by a long tail of far less popular content.
A multi-layer cache handles long-tail content.
Why the tail needs a different tier
The head is easy: a small number of hot objects fit in RAM at every edge and serve almost all traffic. The tail is the problem — collectively it is a large fraction of requests, but each item is requested rarely, so caching all of it at every edge is enormously wasteful.
The hierarchy resolves this by specializing. Edges hold the head in memory. Parents hold a much larger slice — including the tail — on disk, shared across many edges. A tail request costs one regional hop instead of a global one, and no edge has to store an object it will serve twice a month.
This is why proxy servers keep hot data in RAM and cold data on SSD or disk, and why the tiering exists at all. It is the same head/tail split, expressed twice: once inside a node, once across the tree.
What the hierarchy is really protecting against
The tiers are usually justified by the long tail. There is a second reason, and it is the one that matters during an incident.
A popular object expires at the same moment across every edge that holds it. Every one of them misses. Every one of them fetches from origin.
This is the thundering herd, and the CDN makes it sharper rather than softer: the whole point of an edge tier is that origin is sized for the small fraction of traffic that misses, so a synchronized miss is a load spike against a deliberately under-provisioned origin.
Three mechanisms, and they work at different points:
| Mechanism | What it does | Where it acts |
|---|---|---|
| Request coalescing | N concurrent misses for the same object become one upstream fetch; the rest wait on it and are served from the result | Within a single node — the most effective single fix |
| Tiered caching / origin shield | Edges fetch from a parent rather than origin, so 200 edge misses collapse into a handful of parent fetches and then one origin fetch | Between tiers — this is the hierarchy earning its keep |
| TTL jitter | Randomise expiry slightly per node so they do not all expire at the same instant | At the source of the synchronization |
Coalescing and shielding compose, and together they turn a fan-in of thousands into a single origin request. The jitter is the cheapest of the three and addresses the cause rather than the symptom — identical TTLs are what made the misses simultaneous in the first place.
It is the same reasoning as jittered retry backoff: anything that makes many independent actors act on the same schedule will eventually make them act at the same instant.
Serving stale: the availability property people forget
There is one more thing a cache can do that has nothing to do with speed.
When the origin is down, an edge holding an expired copy has a choice: honour the TTL and return an error, or serve the stale copy anyway.
Two related policies are worth naming because they are cheap and frequently omitted:
Serve stale while revalidating. On expiry, return the stale copy immediately and refresh in the background. No user waits for the origin fetch, which removes the latency spike that every TTL expiry otherwise causes — and it also defuses the stampede, since nobody is blocked on the refresh.
Serve stale on error. If the revalidation fails, keep serving the stale copy for a bounded grace period rather than propagating the failure. The origin can be completely down and cached pages keep working.
The trade is explicit and worth stating as a decision: you are choosing to serve data you know is out of date, in exchange for staying up. For a product page or an article that is obviously right; for a price, a balance, or an inventory count it may not be, and the grace period is where you encode how much staleness the content can tolerate.
The general form is worth carrying beyond CDNs: a cache that refuses to serve stale data converts a dependency's outage into your own.
What happens when a node fails
| Failure | Recovery |
|---|---|
| Child proxy fails | DNS directs clients to another child proxy — the routing layer simply stops returning the dead one |
| Parent proxy fails | Each child knows many upper-layer parent servers and switches to another |
| Origin server fails | The origin is a set of servers with hot backups, and content is in a replicated store; if one fails, others take the load |
Every tier has more than one option upstream, which is the redundancy pattern from the Foundations module applied at each level of the tree — no tier has a single parent whose loss orphans it.
Key takeaway
The hierarchy exists because cache misses aggregate. One tier between edges and origin cuts origin load by the fan-out factor, and gives the long tail somewhere to live that isn't every edge's disk. The tiers also absorb the thundering herd: identical TTLs make every edge miss at the same instant against an origin deliberately sized for miss traffic, which request coalescing, tiered shielding and TTL jitter fix at three different points. And a cache can serve stale on error — a cache that refuses to serve stale data converts a dependency's outage into your own.
Interview signal by level
| Level | What a strong answer sounds like |
|---|---|
| L4 | "Edge servers fetch from the origin when they don't have the content." |
| L5 | Adds the tier: "a parent cache layer between edges and origin, so a miss doesn't always reach the source." |
| Staff+ | Quantifies and connects it to the tail: "1,000 edges missing the same object is 1,000 origin requests flat, or 20 with a parent tier — a 50x reduction. It also gives the long tail a home: head in edge RAM, tail on parent disk, so no edge stores what it serves twice a month." |
Next: how a client finds the right edge in the first place.