What a Monitoring System Must Track
In one line: the Foundations module established that healthy nodes cannot tell what happened to a failed one. Monitoring is the machinery that answers that question — and its scope is far wider than "is the process running?"
The twelve signals
| Layer | What the system tracks |
|---|---|
| Process | Critical local process crashes · Resource usage anomalies (CPU, memory, disk, network) in specific processes |
| Server | Overall server health, including load averages and resource consumption · Hardware faults, such as memory failures or disk degradation · Connectivity to critical external services, like network file systems |
| Data center | Data center hardware status, including network switches and load balancers · Power consumption at server, rack, and data center levels · Power events affecting servers, racks, or the data center |
| Network and global | Routing information and DNS status · Network latency within and across data centers · Network status at peering points · Global service health across data centers, such as CDN performance |
Read the bottom half of that list again
Most engineers, asked to design monitoring, produce the first four items: process crashes, CPU, memory, disk. Those are the easy ones because they are visible from inside the machine.
Power draw, peering-point status, and DNS health are the ones that separate a real design from a toy. They matter because they cause failures that look nothing like a crash — a rack browning out, a transit provider degrading, a DNS record going stale — and they are invisible to any agent running on the server.
Naming a few of these unprompted is a strong signal that you have operated infrastructure rather than only written services on top of it.
Not all metrics aggregate the same way
The twelve signals above are a list of what to record. The shape you record each one in decides what questions you can answer later, and getting it wrong is unrecoverable — you cannot compute an answer from data you did not keep.
| Type | What it is | How it aggregates |
|---|---|---|
| Counter | A value that only ever goes up — requests served, bytes written, errors returned | Sums cleanly across hosts and across time. Rate is the derivative |
| Gauge | A value that moves in both directions — memory in use, queue depth, active connections | Averages or maxima across hosts. Summing is usually meaningless |
| Histogram | A distribution recorded as counts per bucket — request latency, payload size | Buckets sum, and percentiles are computed from the summed buckets |
The third row is the one that carries an interview answer, and it is worth doing properly.
You cannot average percentiles
Suppose each of a hundred servers reports its own p99 latency, and you want the fleet's p99. The obvious move is to average the hundred numbers. It is wrong, and not slightly.
A percentile is a position in a distribution, not a quantity. Averaging positions from different distributions is a meaningless operation: it ignores how many observations sat behind each one, and there is no weighting that fixes it in general, because the same p99 can come from wildly different distributions.
The consequence for the design is concrete: if you store a pre-computed p99 per host, the fleet-wide p99 is not recoverable. The information needed to compute it was discarded at the source.
Recording a histogram — counts per latency bucket — keeps the operation an addition, and addition composes. That is what lets one stored series answer "p99 for this service", "p99 for this region", and "p99 for the fleet" from the same data.
The cost is honest and worth naming: a histogram is many series instead of one — a series per bucket per label combination — so it multiplies straight into the cardinality problem that appears later in this chapter. Bucket boundaries also have to be chosen in advance, and a percentile read from buckets is an approximation whose accuracy depends on how well those boundaries match the real distribution.
The rule to carry: record distributions, not summaries. A summary computed too early cannot be un-computed, and averaging percentiles is the most common way monitoring lies to you while looking healthy.
Why the scope is this wide
Each layer fails in a way the layer above cannot detect on its own:
The last two rows are the important ones. A rack losing power takes the monitoring agent with it, so absence of data is the signal. A congested peering point produces no error anywhere in your fleet — every server reports itself healthy while users time out.
That second case is the gray failure from the Foundations module: the system's view of its own health disagrees with what users experience. Monitoring only catches it if it is watching the network path, not just the servers.
What the industry publishes
Cloud providers run exactly this and expose the output publicly:
| Provider | Status page |
|---|---|
| AWS | health.aws.amazon.com/health/status |
| Azure | status.azure.com/en-us/status |
| status.cloud.google.com |
These pages are the customer-facing surface of a monitoring system of exactly this shape — signals collected per region and service, aggregated, and rendered as health state.
Monitoring is what makes your availability target achievable
The Non-Functional Characteristics chapter showed that availability is MTBF / (MTBF + MTTR), and that MTTR is the cheaper lever. It also showed that most of an incident is spent on detect and diagnose, not on fixing.
Monitoring is the system that shrinks those two phases. At 99.99% availability you have about 4.4 minutes of budget per month — so detection measured in minutes has already spent it. That is why this is a building block rather than an operational nicety.
Key takeaway
Monitor across process, server, data center, and network. The signals that are hardest to collect — power, peering, DNS — are the ones that catch failures where every individual server still reports itself healthy. And the shape you record in decides what you can ask later: counters sum, gauges average, and latency must be a histogram — because percentiles are positions in a distribution and averaging them across hosts is meaningless, so a pre-computed per-host p99 makes the fleet-wide p99 unrecoverable.
Interview signal by level
| Level | What a strong answer sounds like |
|---|---|
| L4 | "We'd monitor CPU, memory, and whether the service is up." |
| L5 | Covers the stack: "process health, server resources, hardware faults, plus network latency between data centers." |
| Staff+ | Reaches the invisible failures: "I'd also watch power at rack level and peering-point status, because those produce failures where every server reports healthy and users still time out. Absence of data from a rack is itself a signal." |
Next: the three components that collect and serve it.