Querying, Alerting, and Dashboards

Why this matters: collection is the plumbing; this is the product. A monitoring system nobody trusts — because it pages too often or shows nothing useful — is worse than none, because it consumes attention while providing false assurance.

Key takeaway

The querying service fetches data from the database to support visualization and alerting. It lets users view specific metrics — memory usage on a node — or check if values exceed defined limits.

One query service, two consumers

The two consumers want opposite things, which is worth noticing:

	Alert manager	Dashboard
Reads	Continuously, automatically	On demand, by a human
Query shape	Narrow — one rule, recent window	Broad — many series, long window
Latency need	Low — detection time is MTTR	Tolerant — a human is waiting, not a pager
Output	A notification	A picture
Failure mode	Too many alerts, or none	Too much detail to read

Alert manager

The alert manager monitors metrics against the rules defined in the rules database. If a rule is violated, it triggers notifications via channels like email or Slack.

That is the mechanism. The hard part is deciding what to alert on.

Alert fatigue is the failure mode that matters

A system that pages on every threshold breach trains its team to ignore the pager. Once that happens the monitoring system is actively harmful: it consumes attention, creates false confidence, and the one real alert arrives among fifty that were noise.

Two rules that prevent it:

Alert on symptoms, not causes. Page when users are affected — error rate up, latency up, requests failing. Do not page because one node's CPU hit 90%, which may be entirely fine. CPU belongs on a dashboard for diagnosis; user-visible impact belongs on a pager.
Every page must be actionable. If the recipient's only possible response is "yes, I see it too," it should have been a ticket or a dashboard, not a page.

This connects directly to the Foundations gray-failure lesson: alerting on server-side health metrics catches crashes and misses the degradation users actually feel.

Dashboard

Dashboards visualize collected metrics, providing a high-level view of system health — for example, request counts over the current week.

The distinction from alerting is worth being explicit about:

Alerts    -> "something is wrong, come look"      (push to a human)
Dashboards -> "here is what is happening"         (pull by a human)

Alerts answer whether to investigate. Dashboards answer what is happening once you do. A design that conflates them produces either a wall of unread graphs or a pager that never stops.

Key takeaway

One querying service, two consumers with opposite requirements. Alert on user-visible symptoms with a duration clause; put causes on dashboards for diagnosis. A pager that cries wolf is worse than no pager.

Interview signal by level

Level	What a strong answer sounds like
L4	"We'd send an alert when a threshold is crossed and show graphs on a dashboard."
L5	Separates the two: "alerts for things needing action, dashboards for investigating — and rules need a duration so we don't page on a momentary spike."
Staff+	Names alert fatigue and symptom-based alerting: "page on user-visible symptoms, not causes — CPU at 90% may be fine, a rising error rate never is. Every page must be actionable, or the team learns to ignore the pager and the system becomes worse than nothing."

Next: what breaks when one monitoring server isn't enough.