Logging Within a Server

In one line: this lesson contains the design's most important trade-off, stated plainly: buffering in RAM to keep logging off the critical path means accepting that some logs are lost.

The per-node accumulator

The example: an e-commerce application may run authentication and cart services simultaneously, each producing logs.

We use a unique ID comprising the application-id, service-id, and a timestamp to identify events and determine causality.

The accumulator:

1. Receives the logs.
2. Stores the logs locally.
3. Pushes the logs to a pub-sub system.

Moving the accumulator onto every node is what removes the bottleneck

Lesson 6's accumulator was one service for the fleet — every log line in the system flowed through it.

Here there is one accumulator per server, handling only that server's logs. Its throughput requirement is now bounded by what a single machine can generate, which by definition a single machine can handle.

And it scales for free: add a server, get its accumulator with it. The ratio of accumulators to log volume stays constant no matter how large the fleet grows.

The aggregate volume has not disappeared — it has moved to the pub-sub tier, which is a building block designed for exactly this. That is the move worth naming: push per-node work to the node, and give the aggregate to a component built to absorb it.

Why a local store before the pub-sub push

Step 2 — "stores the logs locally" — is easy to skim past, and it does two jobs.

It is a buffer. The accumulator batches locally and pushes asynchronously rather than making a network call per log line. Lesson 4 established that logging is I/O-heavy; batching is what makes the per-line cost negligible.

It is a fallback. If the pub-sub tier is unreachable, logs still land on local disk and can be shipped when it recovers. That is what keeps a logging outage from becoming a total loss of visibility.

Note the interesting inversion: Lesson 1 said local storage is fatal because logs die with the node. Here local storage is a staging area, not a destination. The distinction is that logs are expected to leave — local disk is where they wait, not where they live.

Pub-sub for throughput

A pub-sub system is used to support high log throughput. Each server's log accumulator publishes events to the pub-sub system, which processes high log volumes. To minimize latency impact, logs are sent asynchronously using a background worker. This prevents logging from degrading application performance or availability.

Pub-sub rather than a queue, because logs have several consumers

The choice matters, and pub-sub gives the reason: a queue message is processed by exactly one consumer; a pub-sub message is read by every subscriber.

Lesson 8 puts three consumers on this stream — a filterer, an error aggregator, and an alert aggregator — plus archival to blob storage. Each needs to see every log, independently.

With a queue, those four would compete: each message would go to exactly one of them, which is useless. Pub-sub gives each its own complete copy, and adding a fifth consumer later requires no change to any producer.

That is exactly the point Lesson 1 made about logs having four different customers with conflicting needs. Pub-sub is what lets each have its own pipeline over the same stream.

Push or pull

This design has agents pushing to the pipeline. The alternative — the pipeline pulling from each node — is a real fork, and Prometheus versus Datadog is the canonical split.

For logs the choice is nearly forced: log lines are events that occur at unpredictable times, not values you can read on demand, and there is nothing to scrape between them. Pull is a good fit for metrics, where the current value always exists and the collector controlling the rate is a feature.

The one thing push gives up is worth naming: silence is ambiguous. A node that stops sending might be idle or might be dead, and the logging system cannot tell. Pull-based metrics answer that for free, which is one more reason the two pillars are usually collected differently even when they land in the same pipeline.

The trade-off

Logging massive volumes of messages involves a trade-off between latency and guarantees of persistence. To minimize latency, services often buffer data in RAM and asynchronously persist it. We can minimize data loss by adding redundant log accumulators to handle the growing number of concurrent users.

Buffering in RAM means a crashing process loses its logs — which is the worst possible time

Follow the failure. A service buffers log lines in memory. The process crashes. Anything not yet flushed is gone.

The cruelty is the correlation: the logs most likely to be lost are the ones immediately preceding a crash — which are exactly the ones explaining the crash. The failure destroys its own evidence.

Three partial mitigations, none free:

• Shorter flush intervals narrow the window and increase I/O — you are trading back the latency you bought.
• Write to local disk first (the accumulator's local store) so a process crash is survivable, though a node failure is not.
• Redundant accumulators, as

But the fundamental trade does not go away, and it is worth stating as a decision rather than a gap: we chose fast, lossy logging over slow, reliable logging, because a logging system that slows the product has failed at its purpose.

Contrast message-queue design, which would not acknowledge a message until it was durably replicated. Different component, opposite priority — and the reason is that a lost message is lost work, while a lost log line is lost visibility.

Backpressure, and the catch-up trap

The pub-sub tier is a buffer, and a buffer that survives an outage is also a debt you have to repay.

Suppose the consumers are down for five minutes. Five minutes of logs are now sitting in the tier, waiting. When the consumers come back they must serve live traffic plus the backlog, and whether they ever catch up is arithmetic:

Consumers running at 50% of capacity
  -> 50% headroom -> a 5-minute backlog drains in ~5 minutes

Consumers running at 75% of capacity
  -> 25% headroom -> the same backlog drains in ~15 minutes

Consumers running at 100% of capacity
  -> zero headroom -> it NEVER drains

The second-order effect is what makes this a logging-specific problem rather than a generic queue problem. A consumer that is fifteen minutes behind is not merely slow — every alert it produces is fifteen minutes stale, so on-call is being paged about a system state that no longer exists, and the dashboard they open to investigate is showing history.

So the honest decision, and it is the opposite of what most components in this course choose: for logging, prefer to drop rather than to lag. Skip the backlog, return to live, and accept a gap in the record. A lost interval is a hole you can see and reason about; a permanently lagging pipeline is a system that lies about the present.

That has two design consequences worth volunteering: size consumers with real headroom rather than at their steady-state ceiling, and make the drop explicit and counted — a logs_dropped counter that itself alerts, so going blind is an event rather than a silence.

Multi-tenant versus single-tenant

"How does logging differ on a multi-tenant cloud (e.g. AWS) versus an organization with exclusive control of its infrastructure (e.g. Facebook)?"

Security might be one aspect that differs. When we encrypt all logs and secure a logging service end-to-end, it does not come for free and incurs performance penalties. Strict log separation is required in a multi-tenant setting, whereas we can improve storage and processing utilization in a single-tenant setting.

Take Meta's Facebook. They have millions of machines generating logs, and the logs can reach several petabytes per hour. Each machine pushes its logs to a pub-sub system named Scribe. Scribe retains data for a few days, and various other systems process the information stored in it. They also store the logs in distributed storage.

For multi-tenancy, we need a separate pub-sub instance per tenant (or per application) for strict separation of logs.

'A separate pub-sub instance per tenant' is expensive — and it is the price of isolation

A software control is one a bug can defeat. A structural one is not.

Strict separation means you cannot simply tag logs with a tenant ID and filter downstream. Filtering is a software control; a bug or a misconfigured query leaks one tenant's logs to another, and Lesson 4 established that logs are a plaintext narrative of everything a system did.

So separation is enforced structurally — separate instances — because that is the only control that survives a software bug.

The cost is real: per-tenant infrastructure means poor utilization, since each instance is provisioned for its tenant's peak rather than pooling across tenants. The source names it: single-tenant deployments can "improve storage and processing utilization" precisely because they need not isolate.

This is the same argument as performance isolation in message-queue design and cells in CDNs: isolation costs efficiency, and you buy it when the blast radius of a leak is unacceptable. For logs containing other customers' data, it is.

Scribe's numbers are worth carrying

Millions of machines. Several petabytes per hour. Retained for a few days.

Two things to take from it. First, it confirms Lesson 6's bottleneck argument with a real figure — no single accumulator absorbs petabytes per hour, which is why Scribe is a pub-sub system rather than a service.

Second, "retains data for a few days" shows the pub-sub tier is transient, not the archive. That matches pub-sub exactly: retention is a capacity decision, and a few days is the window in which consumers must have read. Durable copies go to distributed storage, which Lesson 8 covers.

Note: sensitive applications, such as banking or financial services, require strict security to prevent data theft. A common practice is to encrypt data before logging it, ensuring unauthorized parties cannot decrypt the information.

Interview signal by level

Next: what happens to the stream once it leaves the node.