Scaling with a Hybrid Hierarchy

In one line: the design so far works for one data center and fails for a fleet. The fix — pull locally, push globally — is a general scaling pattern worth carrying beyond monitoring.

What the current design gets right and wrong

	Detail
Pros	Detects signs of impending problems early · Prevents network congestion by controlling data flow via the pull model · Provides high availability for monitoring data
Cons	A dedicated monitoring server is a SPOF — a failover helps, but one server cannot scale indefinitely · Storing high-resolution data indefinitely is resource-intensive, requiring retention policies

Both problems have the same root: everything funnels into one place.

The hybrid hierarchy

We initially used pull to avoid network congestion and decouple applications from monitoring logic. To scale, combine both strategies by level:

Local pull — assign "secondary" monitoring servers to specific clusters, for example one server monitoring 5,000 nodes. These pull data from local nodes.
Global push — secondary servers push aggregated data to a primary data center server, which pushes to a global monitoring service responsible for all data centers.

The hierarchy distributes the load. Blob storage holds bulk data, Elasticsearch provides search, and a visualizer handles analytics. As servers increase, simply add more secondary monitoring nodes.

Why pull locally and push globally

The two levels have genuinely different constraints, and each strategy fits one of them.

Locally, targets are numerous, close, and churning. Pull wins: the collector controls the rate, and a failed scrape tells you a node is gone. Network cost within a cluster is cheap.

Globally, the links are long, expensive, and shared. Pull would mean a global server reaching across the WAN to thousands of secondaries — enormous fan-out over the most constrained links. Push wins: each secondary sends aggregated data upward, so what crosses the WAN is a summary rather than raw samples.

The general rule: pull where fan-out is high and links are cheap; push where fan-out is high and links are expensive.

What if a monitoring system goes down?

We can store the data locally and wait for the system to be back up. But there's a limit to local storage. So either we delete previous data, or we don't store new data. To decide, relevant policies need to be created.

That is an honest statement of a genuine dilemma. When the upstream is unreachable, a secondary buffers — and buffers are finite:

Choice when the buffer fills	Consequence
Delete previous data	Keep the most recent — you lose history but retain what's happening now
Stop storing new data	Keep the older window — you lose visibility into the ongoing incident

Monitoring the thing you depend on

The sharpest question in this chapter: how can a monitoring system work reliably if it runs on the same infrastructure it monitors? A data center network failure can knock out the monitoring components along with everything else.

The answer, per the design:

Mitigation	Detail
Isolated network	An internal, monitoring-specific network separate from the common network
Separate instances	A separate instance of blob stores and other services, not shared with production
External components	Components external to the monitored environment — potentially an independent service provider's infrastructure

However, designing such a system is complex and more expensive.

This is the circular-dependency problem, and it's the best deep-dive in the chapter

The watchdog does not need to be sophisticated — it needs to fail independently. A handful of signals is enough: is the pipeline accepting writes, how far behind is it, and is anything at all arriving from each region.

If monitoring shares fate with the thing it monitors, it goes dark exactly when you need it — and worse, the silence is ambiguous. Is the fleet healthy and quiet, or is everything down including the observer?

Full isolation is expensive, so the realistic posture is tiered: run the bulk of monitoring on shared infrastructure, and keep a small, independent external check — a third-party prober hitting your service from outside — as the answer to "is anything alive at all?"

That external check is cheap, and it is the one signal you can trust when your own data center stops reporting. This is the failure-independence principle from the Foundations module applied to the observer itself.

Key takeaway

Pull locally, push globally, aggregate at each level. And keep at least one observer outside the system it watches — otherwise silence is indistinguishable from health.

Interview signal by level

Level	What a strong answer sounds like
L4	"Add more monitoring servers as the fleet grows."
L5	Describes the hierarchy: "secondary servers pull from local clusters and push aggregates up to a per-DC primary, then to a global service."
Staff+	Explains the pull/push split and the circular dependency: "pull locally because links are cheap and a failed scrape is a signal; push globally because the WAN can't take that fan-out, and we send aggregates not raw samples. And I'd keep an external prober outside our infrastructure — otherwise a data center failure takes out the monitoring too, and silence becomes ambiguous."

Next: showing millions of servers on one screen.