Logging at the Data Center Level

In one line: this is where pub-sub earns its place. Three independent consumers read the same stream, each doing something different with every log line — which a queue could not support.

The architecture

Data in the pub-sub system is transient and is eventually moved to archival storage. While data is available there, three services process it:

Three consumers over one stream is why this is pub-sub, not a queue

This is the concrete payoff of Lesson 7's choice. Every log line must be seen by all three services — the filterer to route it, the error aggregator to check whether it is an error, the alert aggregator to check whether it is fatal — plus archived to blob storage.

Under a queue, each message goes to exactly one consumer. The filterer would get a third of the logs, the error aggregator a third, the alert aggregator a third, and each would be blind to the rest. Errors would go unnoticed two times in three.

Under pub-sub, each consumer has its own offset over the same immutable stream and reads everything independently. A slow filterer does not delay alerting, and adding a fourth consumer later requires no change to any producer.

That is pub-sub's central property — "adding a consumer duplicates the work rather than splitting it" — being exactly what this design needs.

The error and alert aggregators convert logging from pull to push

Lesson 6 noted the initial design had no alerting: logs landed in storage and waited to be searched. That is pull — it only helps someone who already knows to look.

These two consumers make part of the pipeline push. A fatal error does not wait for a human to run a query; it notifies stakeholders and monitoring tools as it flows through.

The distinction matters operationally: searching logs shortens an investigation; alerting on logs starts one. MTTR from Lesson 1 includes time-to-detection, and no amount of searchability improves that if nobody knows to search.

Note the split between the two: the error aggregator notifies the client — the application that produced the error — while the alert aggregator notifies stakeholders and monitoring for fatal errors. Different audiences, different urgency.

The filterer is what makes multi-tenancy and per-application policy possible

"Stores logs in the specific blob storage reserved for that application" does more work than it appears.

It is the routing step that gives each application its own storage, which in turn enables:

• Access control per application — a team sees its own logs, not everyone's.
• Per-application retention — Lesson 8's expiration checker can apply different rules per store, which matters because Lesson 1's consumers want different retention.
• Multi-tenant separation at the storage layer, complementing Lesson 7's per-tenant pub-sub instances.

It works because the unique_ID from Lesson 5 carries the application-id — the filterer reads it and routes. That is the field earning its place in the ID.

Note this is a software control at the storage layer, which is why Lesson 7 argued strict tenant isolation also needs structural separation upstream. Defence in depth: separate instances and separate stores.

What the alert aggregator actually has to do

"Notifies stakeholders of fatal errors" is one line of specification hiding the component that most often gets a logging system switched off.

Consider what naive alerting produces. A shared dependency fails; 100 service instances each log a fatal error; the aggregator fires 100 notifications. On-call's phone buzzes 100 times for one incident. And because the aggregator re-evaluates continuously, the same condition fires again on the next pass, and the next.

The result is alert fatigue — and alert fatigue is not an annoyance, it is an availability problem, because the alert that mattered arrives in a stream nobody is reading any more.

Four mechanisms, each fixing a specific failure:

This is why real systems separate evaluating the condition from managing the notification — Prometheus and its Alertmanager being the clearest example. They are different problems: evaluation is a data question that scales with log volume, notification is a state machine per alert that scales with incident count. Keeping them in one component means a notification outage stops evaluation, which is the wrong coupling entirely.

It also closes a loop opened earlier in the chapter. Severity levels decay because engineers escalate their own messages to be seen; dedup and grouping are what make a noisy ERROR level survivable, and they are the mechanism behind the rule that ERROR should mean "a human should look at this."

Expiration and cold storage

We introduce an expiration checker responsible for:

• Verifying logs for deletion.
• Moving logs to cold storage.

"Do we store the logs permanently?"

Logs also have an expiration date. We can delete regular logs after a few days or months. Compliance logs are usually stored for 3 to 5 years. It depends on the application's requirements.

Two different retention policies, and they map onto Lesson 1's conflicting consumers

Days-to-months for regular logs; 3 to 5 years for compliance logs. That is a difference of two orders of magnitude, and it is not arbitrary.

Debugging logs are only useful while the incident is recent — nobody investigates last quarter's latency spike, so retaining them is pure cost.

Compliance logs must exist because a regulator or auditor may ask, and "we deleted it" is not an acceptable answer.

Lesson 1 flagged these consumers as having conflicting requirements, and this is the resolution: different log classes get different retention, which is only possible because the filterer already routed them into separate stores.

The cold storage move is object storage's tiering, applied. Compliance logs are written once and read approximately never, so archive-tier storage — cheap, slow, rehydration measured in hours — is exactly right. Waiting hours to satisfy an audit request is fine; paying hot-tier prices for five years of untouched data is not.

Expiration is a compliance obligation in both directions

The expiration checker deletes, and deletion is legally mandated both ways:

Data you must delete. Privacy regimes cap how long personal data may be retained. If logs contain anything user-linked, a deletion deadline is a legal requirement and the expiration checker is how it is met at scale.

Data you must keep. Financial and medical records carry minimum retention. Deleting a compliance log at three years when the requirement is five is a violation.

So a single component enforces "delete this by law" and "you may not delete this by law", distinguished only by which store the filterer routed the logs into. That makes the filterer's routing decision legally significant, not just operationally convenient — and mis-routing is a compliance incident rather than an inconvenience.

An alternative: Windows Azure Storage

Note: Windows Azure Storage (WAS) uses an extensive logging infrastructure. Instead of pushing massive volumes of logs to distributed storage, they store logs on local disks and use a grep-like utility for distributed search. This provides a unified view of globally distributed log data without the overhead of centralizing all raw logs.

WAS inverts the design — move the query to the data, not the data to the query

This is a genuinely different architecture and worth understanding as a contrast.

This chapter's design: ship all logs to a central store, index them, query the index. Move the data.

WAS: leave logs where they were written, and run a distributed grep across the fleet when you need to query. Move the query.

The trade is clear. WAS avoids the enormous cost of centralizing petabytes that will mostly never be read — and Lesson 8 noted most log lines are never read by anyone. But queries are slower (fan-out to every node, no index) and logs are not durable against node loss, which was Lesson 1's original complaint.

It is the right choice when the ratio of logs-written to logs-read is extreme, which at storage-system scale it is. And it is the same principle as the inverted-index pattern's document partitioning: when data volume dwarfs query volume, take the computation to the data.

Mentioning this in an interview shows you know the centralized pipeline is a choice with a cost, not the only way.

Interview signal by level

Next: stitching one request back together.