Scaling with a Hybrid Hierarchy
Why this matters: the design so far works for one data center and fails for a fleet. The fix — pull locally, push globally — is a general scaling pattern worth carrying beyond monitoring.
Key takeaway
A dedicated monitoring server creates a SPOF, and while a failover server improves reliability, a single server cannot scale indefinitely as monitored nodes increase. The answer is a hybrid approach combining pull and push in a hierarchy.
What the current design gets right and wrong
| Detail | |
|---|---|
| Pros | Detects signs of impending problems early · Prevents network congestion by controlling data flow via the pull model · Provides high availability for monitoring data |
| Cons | A dedicated monitoring server is a SPOF — a failover helps, but one server cannot scale indefinitely · Storing high-resolution data indefinitely is resource-intensive, requiring retention policies |
Both problems have the same root: everything funnels into one place.
The hybrid hierarchy
We initially used pull to avoid network congestion and decouple applications from monitoring logic. To scale, combine both strategies by level:
- Local pull — assign "secondary" monitoring servers to specific clusters, for example one server monitoring 5,000 nodes. These pull data from local nodes.
- Global push — secondary servers push aggregated data to a primary data center server, which pushes to a global monitoring service responsible for all data centers.
The hierarchy distributes the load. Blob storage holds bulk data, Elasticsearch provides search, and a visualizer handles analytics. As servers increase, simply add more secondary monitoring nodes.
What if a monitoring system goes down?
We can store the data locally and wait for the system to be back up. But there's a limit to local storage. So either we delete previous data, or we don't store new data. To decide, relevant policies need to be created.
That is an honest statement of a genuine dilemma. When the upstream is unreachable, a secondary buffers — and buffers are finite:
| Choice when the buffer fills | Consequence |
|---|---|
| Delete previous data | Keep the most recent — you lose history but retain what's happening now |
| Stop storing new data | Keep the older window — you lose visibility into the ongoing incident |
Monitoring the thing you depend on
The sharpest question in this chapter: how can a monitoring system work reliably if it runs on the same infrastructure it monitors? A data center network failure can knock out the monitoring components along with everything else.
The answer, per the source:
| Mitigation | Detail |
|---|---|
| Isolated network | An internal, monitoring-specific network separate from the common network |
| Separate instances | A separate instance of blob stores and other services, not shared with production |
| External components | Components external to the monitored environment — potentially an independent service provider's infrastructure |
However, designing such a system is complex and more expensive.
Key takeaway
Pull locally, push globally, aggregate at each level. And keep at least one observer outside the system it watches — otherwise silence is indistinguishable from health.
Interview signal by level
| Level | What a strong answer sounds like |
|---|---|
| L4 | "Add more monitoring servers as the fleet grows." |
| L5 | Describes the hierarchy: "secondary servers pull from local clusters and push aggregates up to a per-DC primary, then to a global service." |
| Staff+ | Explains the pull/push split and the circular dependency: "pull locally because links are cheap and a failed scrape is a signal; push globally because the WAN can't take that fan-out, and we send aggregates not raw samples. And I'd keep an external prober outside our infrastructure — otherwise a data center failure takes out the monitoring too, and silence becomes ambiguous." |
Next: showing millions of servers on one screen.