Storage: Time-Series, Blob, and Rules

Why this matters: "store the metrics" hides three genuinely different workloads with three different access patterns. Separating them is what keeps the hot path fast and the retention bill affordable.

Key takeaway

The system requires three distinct storage mechanisms: a time-series database for fast local reads and writes, blob storage for long-term retention, and a rules database holding alert configurations and their actions.

The three stores

Store	Holds	Why separate
Time-series database (TSDB)	Metrics, stored locally on the monitoring server	Fast write and read operations — this is the hot path
Blob storage	Metric data for long-term retention	A separate storage node; cheap capacity, slow access, unbounded size
Rules database	Alert configurations and their corresponding actions	Small, rarely written, read by the alert manager — a different workload entirely

The rules database

Worth dwelling on, because it is the component that turns a passive data store into a system that does something.

For example, if CPU usage exceeds 90%, the system must trigger an alert to the administrator. This requires a rules database to store rules and their corresponding actions.

So a rule is a pair:

condition                     ->  action
CPU usage > 90% for 5 min     ->  page the on-call engineer
Disk free < 10%               ->  open a ticket
Error rate > 1% for 2 min     ->  post to the incident Slack channel

The hot/cold split

The TSDB is described as local to the monitoring server and the blob store as a separate node. That is the same head/tail split the CDN chapter used, applied to time rather than popularity:

	Time-series database	Blob storage
Data age	Recent — minutes to days	Historical — weeks to years
Query frequency	Constant — dashboards, alerts	Rare — post-incident review, capacity planning
Latency requirement	Low — alerts depend on it	Tolerant
Cost per byte	High	Low
Resolution	Full	Often downsampled

Key takeaway

Three stores, three workloads: fast and recent, cheap and historical, small and configurable. The retention policy between the first two is a first-class design decision, not an operational afterthought.

Interview signal by level

Level	What a strong answer sounds like
L4	"Store the metrics in a database."
L5	Splits the tiers: "a TSDB for recent data, blob storage for long-term retention, and somewhere to keep alert rules."
Staff+	Prices retention and names the trap: "per-second data forever is unaffordable at fleet scale, so I'd downsample by age — seconds for a day, minutes for a month, hours for a year. And I'd cap label cardinality, because unbounded labels are the standard way a monitoring system kills itself."

Next: the decision that shapes the whole architecture.