Cheat Sheet

Key takeaway

Monitoring shrinks detect and diagnose — the two phases that dominate MTTR. Collect with pull for rate control, store in three tiers, scale with a local-pull/global-push hierarchy, alert on symptoms, and visualize with heat maps sorted by topology.

What to track

Layer	Signals
Process	Critical process crashes · resource anomalies (CPU, memory, disk, network) per process
Server	Overall health and load averages · hardware faults (memory failure, disk degradation) · connectivity to external services (network file systems)
Data center	Hardware status (switches, load balancers) · power consumption at server/rack/DC level · power events
Network / global	Routing and DNS status · latency within and across DCs · peering-point status · global service health (CDN)

The forgotten ones — power, peering, DNS — catch failures where every server reports healthy and users still time out. (Gray failure.)

Public examples: AWS, Azure, and Google all publish status pages fed by systems of this shape.

High-level design

Storage (TSDB) · Data collector (fetch and persist) · Querying service (API over the TSDB).

The three stores

Store	Holds	Why separate
Time-series DB	Metrics, local to the monitoring server	Fast writes and reads — the hot path
Blob storage	Long-term retention, separate node	Cheap, slow, unbounded
Rules database	Alert conditions and their actions	Small, rarely written; a config change, not a deploy

Why a TSDB: append-only · time-ordered · write-dominated · queried by range · value declines with age · highly compressible.

Retention is a design decision: per-second for a day → per-minute for a month → per-hour for a year. Storing full resolution forever is unaffordable at fleet scale.

Cardinality is the classic killer: each metric+label combination is its own series. Unbounded labels (user ID, request ID, URL path) explode memory and kill queries. Keep labels low-cardinality.

Pull vs push

Aspect	Pull	Push
Initiates	The monitoring system	Each app/server
Controls the rate	The collector — bounded	Every target independently
Congestion	Controlled	Floods, worst during incidents
Freshness	Scrape-interval bound	Near real time
Target setup	Expose an endpoint	Daemon on every host
Knowing targets	Needs service discovery	Targets announce themselves
Dead target	Failed scrape = explicit signal	Silence is ambiguous

Why push floods: failing services emit more, so monitoring load spikes exactly when infrastructure is struggling — same shape as a retry storm.

Real-world pull at scale: DigitalOcean monitors millions of globally dispersed machines this way.

Collector input: metrics extracted from application logs via a distributed messaging queue (message carries service name, ID, log description) — the queue decouples log production from metric consumption.

Service discovery

Static target lists fail because autoscaled instances run unmonitored. A discovery component integrates with EC2, Kubernetes, or Consul; the collector queries it for scrape targets.

Keep separate: discovery says what should exist; the scrape says what's healthy. Conflating them pages on every rolling deploy.

Querying, alerting, dashboards

Alerts     -> "something is wrong, come look"   (pushed to a human)
Dashboards -> "here is what is happening"       (pulled by a human)

Alert manager — evaluates metrics against the rules DB, notifies via email or Slack. Dashboard — high-level health view (e.g. request counts this week).

Alerting rules:

Alert on symptoms, not causes. Error rate and latency page; a node at 90% CPU does not.
Every rule needs a duration clause. CPU > 90% fires on a GC pause; for 5 minutes fires on a problem.
Every page must be actionable, or the team learns to ignore the pager.

Incident phases: alert manager owns detect; dashboard owns diagnose and verify.

Scaling: the hybrid hierarchy

Nodes  --LOCAL PULL-->  Secondary (~5,000 nodes each)
                          --GLOBAL PUSH (aggregated)-->  Primary (per DC)
                                                           --push-->  Global service
                                                                        -> Blob storage
                                                                        -> Elasticsearch
                                                                        -> Visualizer

Why hybrid: locally, links are cheap and a failed scrape is a signal → pull. Globally, WAN links are expensive and fan-out is huge → push aggregates. Scale by adding secondaries. Hierarchy is a general scalability pattern — same as CDN tiers and LB tiers.

Single-server design fails on: SPOF (failover helps but doesn't scale) · indefinite high-resolution storage.

If the upstream is down: buffer locally, but buffers are finite — drop the oldest, since recent data is what matters during an incident. Write the policy before the incident.

Monitoring the monitor

The circular dependency: if monitoring shares infrastructure with what it monitors, a DC failure takes it out — and silence becomes ambiguous (healthy and quiet, or dead?).

Mitigations: isolated monitoring-specific network · separate instances of blob storage and supporting services · components external to the environment, possibly a third-party provider. Complex and expensive — so realistically, run the bulk internally and keep one cheap external prober as the signal you can trust.

Heat maps

Each cell = one component. Green = normal · red = non-responsive after multiple attempts.

Sort by data center, cluster, and row — the layout is the diagnosis:

scattered red   -> unrelated individual failures (noise)
one row red     -> rack problem (power or ToR switch)
block red       -> shared switch or PDU
region red      -> data center / network event

1 bit per server: 1,000,000 bits / 8 = 125,000 bytes = 125 KB

Small enough for a live view. Extends to filesystems, network switches, and links.

Quick decision cues

Need bounded data rate → pull
Need near-real-time → push (accept the flood risk)
Autoscaling fleet → service discovery, read from the orchestrator
Fleet outgrew one collector → secondary tier, pull locally
Crossing the WAN → push aggregates, not raw samples
Storage growing without bound → check cardinality first, then retention
Team ignoring alerts → alert on symptoms, add duration clauses
Everything green, users unhappy → gray failure; measure client-side
Can't install an agent → black-box probing
Need to trust a signal during a DC outage → external prober

Work the Interview Walkthrough for the full design and the Concept Drills for rapid-fire practice.