Storage: Time-Series, Blob, and Rules

In one line: "store the metrics" hides three genuinely different workloads with three different access patterns. Separating them is what keeps the hot path fast and the retention bill affordable.

The three stores

Store	Holds	Why separate
Time-series database (TSDB)	Metrics, stored locally on the monitoring server	Fast write and read operations — this is the hot path
Blob storage	Metric data for long-term retention	A separate storage node; cheap capacity, slow access, unbounded size
Rules database	Alert configurations and their corresponding actions	Small, rarely written, read by the alert manager — a different workload entirely

The rules database

Worth dwelling on, because it is the component that turns a passive data store into a system that does something.

For example, if CPU usage exceeds 90%, the system must trigger an alert to the administrator. This requires a rules database to store rules and their corresponding actions.

So a rule is a pair:

condition                     ->  action
CPU usage > 90% for 5 min     ->  page the on-call engineer
Disk free < 10%               ->  open a ticket
Error rate > 1% for 2 min     ->  post to the incident Slack channel

The hot/cold split

The TSDB is described as local to the monitoring server and the blob store as a separate node. That is the same head/tail split CDNs used, applied to time rather than popularity:

	Time-series database	Blob storage
Data age	Recent — minutes to days	Historical — weeks to years
Query frequency	Constant — dashboards, alerts	Rare — post-incident review, capacity planning
Latency requirement	Low — alerts depend on it	Tolerant
Cost per byte	High	Low
Resolution	Full	Often downsampled

Retention is not optional — it's a design requirement

Storing high-resolution data indefinitely is resource-intensive. The system requires retention policies to periodically delete or downsample old data.

The arithmetic is unforgiving. Ten metrics per second per server, across 100,000 servers, is a million samples per second — billions per day, forever, growing with the fleet. No storage budget survives that at full resolution.

Downsampling is the usual answer: keep per-second data for a day, per-minute for a month, per-hour for a year. You lose the ability to see a two-second spike from last March, which is almost always an acceptable trade — and stating the tiers explicitly is what makes it a decision rather than an oversight.

Cardinality, and how to actually stop it

A TSDB stores each unique combination of metric name and labels as its own series. Add a label with unbounded values — a user ID, a request ID, a full URL path — and you create a new series per distinct value.

This is cardinality explosion, and it is the most common way a monitoring system falls over — taking out the infrastructure it was watching, at the moment that infrastructure was already in trouble.

Two properties make it nastier than an ordinary capacity problem. It grows multiplicatively: adding one label with a hundred values multiplies the series count rather than adding to it. And it is usually introduced by someone else — an application team adding a label they consider obviously useful, with no visibility into what it costs the platform.

The usual advice — keep labels low-cardinality, status code yes and user ID no — is correct and insufficient, because a convention is not an enforcement. A design answer needs a mechanism:

Two pieces of state make that work. A policy store mapping each metric name to its permitted label keys and a maximum series count — which can live alongside the alert rules, since it is the same kind of small, read-heavy, admin-edited configuration. And a fast series tracker, typically a set per metric in a key-value store, answering "have I seen this combination before?" on the ingest path.

The trade is real: a membership check per data point adds latency to the hottest path in the system. Batching the checks, or putting a local Bloom filter in front as a first pass, is the usual mitigation — a negative from a Bloom filter is definitive, so only possible hits need the round trip.

And the judgement worth stating: caps too strict drop useful data; caps too loose do not prevent the problem. Which is why the counter matters as much as the cap — dropping silently is how you discover during an incident that the metric you need has been rejected for a month.

High-cardinality identifiers still have a home; it is just not here. A request ID belongs in a log or a trace, where the storage model expects one record per event rather than one series per distinct value.

Key takeaway

Three stores, three workloads: fast and recent, cheap and historical, small and configurable. The retention policy between the first two is a first-class design decision, not an operational afterthought. And cardinality is the failure mode that actually kills these systems — it grows multiplicatively and is usually introduced by another team, so "keep labels low-cardinality" has to be enforced rather than advised: an allowlist of label keys, a per-metric series cap, and a dropped-series counter that itself alerts, because dropping silently is how you find out during an incident.

Interview signal by level

Level	What a strong answer sounds like
L4	"Store the metrics in a database."
L5	Splits the tiers: "a TSDB for recent data, blob storage for long-term retention, and somewhere to keep alert rules."
Staff+	Prices retention and names the trap: "per-second data forever is unaffordable at fleet scale, so I'd downsample by age — seconds for a day, minutes for a month, hours for a year. And I'd cap label cardinality, because unbounded labels are the standard way a monitoring system kills itself."

Next: the decision that shapes the whole architecture.