Logging in a Distributed System

In one line: the cascade below is the scenario logging exists for — many services fail at once, and only one of them is the cause. Finding which is the whole problem.

Why it gets hard

Modern system architectures often adopt microservices rather than monolithic architectures. A single microservice may run on thousands of instances. Because services are interdependent, a failure in one can cascade to others. Without centralized logging and end-to-end request tracing, identifying the root cause of failures becomes significantly harder.

The failure sequence the design walks through:

1. A client's first point of interaction with the distributed system.
2. A service fails.
3. Another service that depended on it also fails because of it.
4. Another dependent service fails.
5. Similarly, other dependent services fail.

Cascading failure inverts the debugging problem — you get too much evidence, not too little

Error volume runs opposite to causal distance. The service screaming loudest is the one furthest from the fault.

The intuition is that debugging is hard because information is scarce. In a cascade it is the opposite: every service in the dependency chain logs an error, so you have dozens of angry services and no indication which one started it.

Worse, the loudest service is usually the wrong one. The user-facing service at the top of the chain produces the most errors and the most customer complaints — and it is the last one to actually break. The real cause is some leaf service three levels down that logged a modest number of timeouts.

Two things follow, and they shape the rest of the chapter:

Timestamps alone will not resolve it. Across thousands of machines with unsynchronized clocks, "which error came first" is not answerable to the precision you need. That is Lesson 9's problem.

You need to reconstruct one request's path, not read one service's log. Filtering by service gives you a service's errors; filtering by request ID gives you the story. That is why Lesson 9's trace ID is the single most valuable thing in this design.

The framing worth carrying: in a cascade, the question is not "what failed" but "what failed first."

'Thousands of instances' is why per-node access does not scale

Lesson 1 established that local logs die with the node. This adds a second problem: even when nodes survive, there are too many of them.

A microservice on a thousand instances means a single request touched one of those thousand — and you do not know which. Grepping a thousand machines to find the one that handled request abc123 is not a debugging strategy; it is a denial of service against yourself.

So central aggregation is forced twice over: logs must survive the node, and they must be searchable across nodes. That second requirement is why Lesson 5 pulls in the Distributed Search building block — with volumes this large, "search the logs" is itself a distributed search problem.

The monolith comparison is worth making explicitly

It is worth being clear about what specifically got harder, because the answer is not "everything."

In a monolith, a request runs in one process on one machine. Logs land in one file, in order, from one clock. Debugging is grep.

In microservices, that same request becomes a dozen processes on a dozen machines, each with its own file and its own clock, running concurrently with thousands of unrelated requests.

Nothing about the application got harder — the business logic is the same. What changed is that the evidence got scattered and interleaved. Distributed logging is entirely about undoing that: re-collect the scattered pieces (aggregation) and re-order them correctly (causality).

Saying it that way in an interview shows you understand distributed logging as a reconstruction problem rather than a storage problem.

Interview signal by level

Next: keeping the volume manageable.