Interview Walkthrough: Design a Monitoring System

"Design a system to monitor a fleet of servers."

This one rewards operational experience more than most. The mechanisms are simple; the judgment — what to alert on, who controls the data rate, who watches the watcher — is where the signal is.

Step 0 — Scope it before you design

• How many servers, in how many data centers?
• What are we monitoring — just servers, or hardware, power, and network too?
• Metrics only, or logs and traces as well?
• How fresh must the data be — seconds, or is a minute fine?
• How long do we retain it, and at what resolution?
• Who consumes it: automated alerting, humans on dashboards, or both?

Then commit:

"I'll assume 100,000 servers across five data centers, monitoring process health, server resources, hardware faults, and network status. Metrics are the focus. Detection within a minute, retention for a year at reducing resolution. Both alerting and dashboards. Let me size the write load first, because that decides the storage design."

Step 1 — A quick BOTEC

Servers                = 100,000
Metrics per server     = 100
Sample interval        = 10 seconds

Samples/sec = 100,000 * 100 / 10       = 1,000,000 samples/sec
Samples/day = 1M * 86,400              = ~86 billion/day

At 16 bytes per raw sample             = ~1.4 TB/day uncompressed
Time-series compression (~10x)         = ~140 GB/day
Per year                               = ~50 TB

"A million samples per second is the number that matters. It's write-dominated, append-only, and time-ordered — which is exactly what a time-series database is built for, and why a relational store would fall over. It also says retention is a real cost: 50 TB a year at full resolution, growing with the fleet. So downsampling isn't optional."

Step 2 — Requirements

What to track, spanning four layers: process crashes and resource anomalies · server health and hardware faults · data center hardware, power draw, and power events · routing, DNS, network latency, peering-point status, and global service health.

"I'd call out the bottom two layers specifically. Power and peering are the ones that cause failures where every individual server reports itself healthy and users still time out — that's the gray-failure case, and you only catch it if you're watching the network path rather than just the hosts."

Step 3 — High-level design

Three components: collect, store, query. Then grow each one.

Step 4 — Storage, split three ways

"Three stores, three workloads. A time-series database local to the monitoring server for recent data on the hot path. Blob storage as a separate node for long-term retention — cheap, slow, unbounded. And a rules database holding alert conditions and their actions, so changing a threshold is a config update rather than a deploy. That last one matters during an incident, when you don't want to ship code to silence a noisy alert.

Retention is a design decision, not an operational one: per-second for a day, per-minute for a month, per-hour for a year. Otherwise 50 TB a year compounds forever."

Step 5 — Collection: the key decision

"Pull, and here's why. The collector controls the scrape interval, so the data rate is bounded by design — if it's overloaded it scrapes less often and degrades gracefully.

Push inverts that. Every service decides independently, and failing services emit more — so monitoring traffic spikes exactly when the infrastructure is already struggling. That's the same self-amplifying shape as a retry storm. Push also needs a daemon installed on every target.

The other advantage people miss: under pull, a failed scrape is an explicit signal that a target is gone. Under push, silence is ambiguous — dead, or just nothing to report? Since detecting dead nodes is the whole point, that's a real argument."

Then add discovery:

"Pull needs to know what to scrape, and in an autoscaling fleet the target list changes constantly. So a service discovery component integrated with Kubernetes, EC2, or Consul — read the target list from the orchestrator that already owns it rather than maintaining a parallel registry."

Step 6 — Scaling

"One monitoring server is both a SPOF and a ceiling. The fix is a hybrid hierarchy:

Local pull — secondary monitoring servers each cover a cluster, say 5,000 nodes, pulling from local targets. Global push — those secondaries push aggregated data to a per-data-center primary, which pushes to a global service.

The reason for the split is that the two levels have different constraints. Locally, links are cheap and fan-out is high, so pull wins. Globally, links are expensive, so we push summaries instead of having one server scrape thousands across the WAN. Scaling means adding secondaries."

Step 7 — Visualization

"Heat maps. One cell per server, green healthy, red unreachable after multiple failed attempts. One bit per server means a million hosts is 125 KB — small enough to be a live view.

The important part is the sort order: by data center, cluster, and row. Then a red row means a rack problem, a red block means a shared switch, and scattered red is just background noise. The pattern gives you the diagnosis before you read a metric."

Deep Dives & Follow-up Questions

"How does the monitoring system work if the data center it's in goes down?"

This is the circular dependency, and it's the sharpest problem here. If monitoring shares fate with what it monitors, it goes dark exactly when needed — and the silence is ambiguous: healthy and quiet, or everything dead including the observer? Mitigations are an isolated monitoring-specific network, separate instances of blob storage and supporting services rather than production's, and ideally components external to the environment, potentially on an independent provider. Full isolation is expensive, so realistically I'd run the bulk on shared infrastructure and keep one cheap external prober hitting us from outside — that's the single signal I can trust when our own data center stops reporting.

"A secondary can't reach the primary. What does it do with the data?"

Buffer locally and wait. But buffers are finite, so eventually it's either drop the oldest or stop accepting new. I'd drop the oldest — during an outage the most recent data is what you need to understand what's happening now, and a frozen picture from before the incident is much less useful. That's a policy decision worth writing down before the incident, not during.

"Your alerts are firing constantly and the team ignores them. Diagnose."

Alert fatigue, and it makes the system worse than nothing — it consumes attention and creates false confidence. Two fixes. Alert on symptoms, not causes: page when users are affected — error rate, latency, failed requests — not because one node's CPU hit 90%, which may be entirely normal. Causes belong on dashboards for diagnosis. And every rule needs a duration clause: "CPU > 90%" fires on a GC pause; "for five minutes" fires on a real problem. If a page's only possible response is "yes, I see it," it should have been a ticket.

"The monitoring system is using more storage than production. What do you do?"

Check cardinality first. A TSDB stores each unique metric-plus-label combination as its own series, so a label with unbounded values — user ID, request ID, full URL path — creates a series per distinct value. That's cardinality explosion and it's the usual culprit. Keep labels low-cardinality; high-cardinality identifiers belong in logs or traces. Then enforce downsampling tiers so full-resolution data ages out.

"Everything is green but customers are complaining. What did you miss?"

Gray failure — we're measuring what the servers think of themselves rather than what users experience. Server-side health checks catch crashes and miss degradation: a node passing its probe while dropping a third of requests looks perfectly healthy. The fix is measuring client-observed success rate and latency, which is exactly why client-side monitoring is a separate concern from everything in this design.

"How would you monitor something you can't install an agent on — a managed database, a third-party API?"

Black-box monitoring: probe it the way a user would and measure the result. You lose internal visibility — no CPU, no memory — but you gain the thing that actually matters, whether it works from the outside. In practice you want both: white-box for diagnosis where you control the host, black-box for detection everywhere.

Now do it live

The next section drills these as standalone probes.