Querying, Alerting, and Dashboards

In one line: collection is the plumbing; this is the product. A monitoring system nobody trusts — because it pages too often or shows nothing useful — is worse than none, because it consumes attention while providing false assurance.

One query service, two consumers

The two consumers want opposite things, which is worth noticing:

Alert manager

The alert manager monitors metrics against the rules defined in the rules database. If a rule is violated, it triggers notifications via channels like email or Slack.

That is the mechanism. The hard part is deciding what to alert on.

Alert fatigue is the failure mode that matters

A system that pages on every threshold breach trains its team to ignore the pager. Once that happens the monitoring system is actively harmful: it consumes attention, creates false confidence, and the one real alert arrives among fifty that were noise.

Two rules that prevent it:

• Alert on symptoms, not causes. Page when users are affected — error rate up, latency up, requests failing. Do not page because one node's CPU hit 90%, which may be entirely fine. CPU belongs on a dashboard for diagnosis; user-visible impact belongs on a pager.
• Every page must be actionable. If the recipient's only possible response is "yes, I see it too," it should have been a ticket or a dashboard, not a page.

This connects directly to the Foundations gray-failure lesson: alerting on server-side health metrics catches crashes and misses the degradation users actually feel.

What the alert manager has to do besides evaluate

"Fires a notification when a rule is breached" is one line of specification hiding the component that most often gets a monitoring system muted.

A shared dependency fails and a hundred instances breach the same rule simultaneously. Naively that is a hundred notifications — and because rules are re-evaluated continuously, the same condition fires again on the next cycle, and the one after.

Each mechanism prevents a specific failure. Deduplication stops an ongoing condition paging every cycle. Grouping stops one cause producing one page per instance. Silencing stops planned maintenance waking someone. Escalation stops an unacknowledged page disappearing quietly.

It pairs with the duration requirement: a rule needs "above threshold for five minutes" rather than a bare threshold, and dedup handles what happens after those five minutes — otherwise a genuinely sustained problem becomes a page per evaluation for its whole duration.

The structural point is that evaluating a condition and managing a notification are different problems, and real systems separate them — Prometheus and its Alertmanager being the clearest example. Evaluation scales with metric volume; notification scales with incident count. Keeping them in one component means a notification-channel outage stops evaluation, which is exactly the wrong coupling.

Dashboard

Dashboards visualize collected metrics, providing a high-level view of system health — for example, request counts over the current week.

The distinction from alerting is worth being explicit about:

Alerts    -> "something is wrong, come look"      (push to a human)
Dashboards -> "here is what is happening"         (pull by a human)

Alerts answer whether to investigate. Dashboards answer what is happening once you do. A design that conflates them produces either a wall of unread graphs or a pager that never stops.

Interview signal by level

Next: what breaks when one monitoring server isn't enough.