Concept Drills: 14 Monitoring Probes

Monitoring rarely appears as the whole question. It appears as "how would you know if this broke?" midway through another design — which is when a crisp answer counts most.

What and why

1. What should a fleet monitoring system track? · L5 · Testing: scope

Four layers. Process — crashes and per-process CPU, memory, disk, network anomalies. Server — load averages, resource consumption, hardware faults like memory failures and disk degradation, connectivity to external services. Data center — switch and load balancer status, power consumption and power events at server, rack, and DC level. Network and global — routing and DNS status, latency within and across data centers, peering-point status, and global service health like CDN performance.

2. Which of those do people forget, and why does it matter? · Staff · Testing: operational depth

Power, peering points, and DNS. They matter because they produce failures where every individual server reports itself healthy and users still time out — a rack browning out, a transit provider degrading. No agent inside the fleet sees them. That's the gray-failure case, and you only catch it by watching the network path rather than the hosts.

3. Why is monitoring a building block rather than an ops concern? · L5 · Testing: connecting to NFRs

Because availability is MTBF / (MTBF + MTTR) and most of an incident is spent on detect and diagnose, not fixing. Monitoring is what shrinks those. At 99.99% you have about 4.4 minutes of budget a month — detection measured in minutes has already spent it, so detection speed is an architectural constraint.

Storage

4. Why a time-series database rather than a relational one? · L5 · Testing: data shape

Metric data is append-only and time-ordered, overwhelmingly write-dominated, queried by time range rather than by key, highly compressible because consecutive samples barely differ, and declines in value with age. TSDBs exploit all of that — delta compression on write, and automatic downsampling or expiry as data ages. A general-purpose database treats every row as equally valuable forever, which is exactly wrong here.

5. What three stores does this design need? · L5 · Testing: recall with reasoning

A time-series database local to the monitoring server for fast reads and writes on recent data. Blob storage as a separate node for long-term retention — cheap and slow. And a rules database holding alert conditions and their corresponding actions, like "CPU above 90% triggers an alert." Three different workloads, so three stores.

6. Why keep alert rules in a database instead of in code? · Staff · Testing: operational instinct

Because thresholds change constantly — during incidents, after capacity changes, when a service's baseline shifts. In a database, changing one is a config update. In code, it's a deploy. You very much do not want "silence this noisy alert" to require shipping a binary while an incident is running.

7. Your monitoring storage is growing without bound. What's the likely cause? · Staff · Testing: the classic failure

Cardinality explosion. A TSDB stores every unique combination of metric name and labels as its own series, so a label with unbounded values — user ID, request ID, full URL path — creates a new series per value. Memory grows without limit and queries crawl. Keep labels low-cardinality: status code yes, user ID no. High-cardinality identifiers belong in logs or traces. Then enforce downsampling by age.

Collection

8. Pull or push, and why? · Staff · Testing: the central decision

Pull, primarily because the collector controls the rate. If it's overloaded it scrapes less often and degrades gracefully. Under push, every service decides independently — and failing services emit more, so monitoring traffic spikes exactly when the infrastructure is struggling. Same self-amplifying shape as a retry storm. Push also needs a daemon on every target.

9. What's push actually good for? · L5 · Testing: fairness

Freshness — it can be near real time, because a service reports the moment something happens rather than waiting to be scraped. It also removes the need for service discovery, since targets announce themselves. That's why real systems use both, at different levels.

10. Under pull, how do you know a server died? · Staff · Testing: an underrated advantage

The scrape fails, which is an explicit, timestamped event you can alert on — absence of data becomes a positive signal. Under push, a target that stops sending is indistinguishable from one with nothing to report, so you'd need a separate liveness mechanism. Since detecting dead nodes is much of the point, that's a real argument for pull.

11. Why does pull need service discovery? · L5 · Testing: the dependency

Because it has to know what to scrape, and in an autoscaling fleet the target list changes by the minute. A static list means new instances run unmonitored — precisely the capacity added during a spike. So a discovery component integrated with Kubernetes, EC2, or Consul, read from the orchestrator that already owns the answer rather than maintained separately.

Scale and alerting

12. Describe the hybrid hierarchy and why it's hybrid. · Staff · Testing: the pattern

Secondary monitoring servers each cover a cluster — say 5,000 nodes — and pull locally. They push aggregated data to a per-data-center primary, which pushes to a global service. It's hybrid because the two levels have different constraints: locally, links are cheap and a failed scrape is a signal, so pull wins. Globally, WAN links are expensive and fan-out would be enormous, so pushing summaries wins. Scale by adding secondaries.

13. Your team ignores the alerts. What went wrong? · Staff · Testing: alert fatigue

The system pages on causes instead of symptoms. Alert when users are affected — error rate, latency, failed requests — not because a node's CPU hit 90%, which may be fine. Causes belong on dashboards for diagnosis. Every rule also needs a duration clause: "CPU above 90%" fires on a GC pause, "for five minutes" fires on a real problem. And every page must be actionable — if the only response is "yes, I see it," it should have been a ticket.

14. How can monitoring be reliable if it runs on the infrastructure it monitors? · Staff · Testing: the circular dependency

It can't, fully — a data center network failure takes the monitoring with it, and then silence is ambiguous: healthy and quiet, or dead including the observer? Mitigations are an isolated monitoring-specific network, separate instances of blob storage and supporting services, and components external to the environment, potentially on an independent provider. That's complex and expensive, so realistically I'd run the bulk on shared infrastructure and keep one cheap external prober outside — the one signal I can trust when our own data center stops reporting.

Self-check

The cheat sheet next compresses the chapter onto one page.