Why Logging
In one line: logging looks like the least architectural topic in this course. It is not — in a distributed system, logs are the only record of what actually happened, and getting them wrong means outages you cannot explain.
Where logging sits
Observability has three pillars, and knowing which one answers which question is the first thing to say.
Metrics tell you that the error rate rose. Logs tell you what the errors were. Traces tell you which service caused them.
They are usually built as one pipeline because they share the collection and transport problem, but they are not substitutes. A metric cannot tell you which user was affected; a log cannot tell you the p99 latency without scanning everything. This chapter builds the log pipeline, and picks up the trace in the last lesson because the two share a request ID.
The need
Logging is critical for understanding event flow in distributed systems. When failures or security breaches occur, logs help identify the root cause and reduce the mean time to repair.
Mean time to repair is the metric logging actually moves
The Non-Functional Characteristics chapter framed availability as a budget. MTTR is the term logging attacks directly.
Availability is roughly a function of how often you break and how fast you recover. You cannot drive failure frequency to zero — hardware dies, deploys go wrong, dependencies fail. So the practical lever is recovery speed, and recovery starts with understanding what happened.
Without logs, an incident begins with hours of guessing. With good logs, it begins with reading. That is the whole value proposition, and it is why logging earns a place among the building blocks despite not serving a single user request.
The corollary is worth stating: logging pays off only during failures. It is pure cost in the happy path, which is exactly why it gets under-invested in until the first bad outage.
Why print statements are not enough
Simple print statements are not suitable for production environments. They do not support severity levels (e.g. INFO or ERROR) and usually write to standard output rather than a persistent log store. Distributed systems generate high log volumes, so logs must be structured and aggregated centrally for efficient analysis.
| Problem with print | Consequence |
|---|---|
| No severity | Cannot filter noise from signal — everything is equally important, so nothing is |
| No structure | Cannot be parsed or queried programmatically; every consumer writes a bespoke parser |
| Hard to track | Writes to standard output on one node — not persistent, not centralized, not correlatable |
The third failure is the one that actually kills you
No severity and no structure are annoying. Writing to local standard output is fatal, and for a reason specific to distributed systems.
If a node's logs live on that node, then debugging a request that touched twelve services means SSHing into twelve machines — assuming they still exist. Under autoscaling or containers they often do not: the instance that produced the interesting log was terminated an hour ago, and its stdout went with it.
The log you most need is the one from the node that died. That is precisely the log a local, non-persistent mechanism guarantees you cannot have.
So central aggregation is not a convenience feature. It is what makes logs survive the failure they are supposed to explain — and it is why the entire rest of this chapter is about getting logs off the node.
What log analysis supports
| Scenario | What it means |
|---|---|
| Troubleshooting | Application, node, or network issues |
| Compliance | Adhering to internal security policies and external compliance regulations |
| Breach response | Detecting and responding to data breaches |
| Product analytics | Analyzing user actions to inform features like recommender systems |
These four have conflicting requirements — which is why one log pipeline rarely serves all of them well
Read them as four different customers, because they want opposite things:
Troubleshooting wants high detail, short retention. You need everything about the last few hours; last quarter is irrelevant.
Compliance wants low detail, long retention — often 3 to 5 years, as Lesson 8 notes — and immutability, since a log you can edit proves nothing.
Breach response wants completeness, because the one event you sampled away is the one the attacker used.
Product analytics wants structure and volume, and tolerates sampling happily since it works statistically.
Those conflict directly. Sampling is fine for analytics and unacceptable for breach response. Long retention is required for compliance and wasteful for debugging.
Real systems therefore route different log classes to different pipelines with different retention and sampling policies — which is exactly what Lesson 8's filterer and expiration checker exist to do. Naming this tension is a strong interview observation, because most candidates treat "logs" as one undifferentiated stream.
Causality is the distributed-systems-specific requirement
Services running concurrently across multiple nodes require causality information to stitch together the correct event flow.
On one machine, ordering is free — a single clock and a single log file give you the sequence. Across thousands of nodes, there is no global clock, and timestamps from different machines are not reliably comparable. Two events a millisecond apart on different servers may be recorded in the wrong order.
So "what happened, in what order" — trivial in a monolith — becomes a genuine distributed systems problem. Lesson 9 solves it with a sequencer-issued request ID that carries happens-before ordering, which is the Sequencer building block reused.
This is the single thing that makes distributed logging harder than logging.
Security concerns
"What are some security concerns when designing a distributed logging system, and how would you mitigate them?"
Logs are an unusually attractive target, because they are a plaintext record of everything the system did:
| Concern | Mitigation |
|---|---|
| Sensitive data leaking into logs | Never log PII or credentials; encrypt before logging where the data is required. Lesson 4 covers this |
| Logs revealing internal structure | Logs expose application flow and internal logic — restrict access and treat the log store as a production-sensitive system, not a debugging convenience |
| Tampering | An attacker who can edit logs can erase their own traces. Append-only storage and integrity checks make logs evidentiary |
| The logging library itself | Lesson 4's Log4Shell — the logging dependency is attack surface. Patch and audit it like any other |
| Multi-tenant leakage | Strict log separation per tenant, covered in Lesson 7 |
Key takeaway
Logs are one of three pillars — metrics say that it broke, logs say what happened, traces say where the time went — and they are not substitutes. Logging exists to reduce MTTR: you cannot stop failing, so you optimize for explaining. Print statements fail on severity, structure, and above all persistence — the log you need most is on the node that died. And logging's four consumers — troubleshooting, compliance, breach response, analytics — want conflicting detail, retention, and sampling.
Interview signal by level
| Level | What a strong answer sounds like |
|---|---|
| L4 | "We need logs to debug problems, and print statements aren't good enough." |
| L5 | Names why centralization matters: "logs on local stdout die with the node, and a request touching twelve services means twelve machines — so logs have to be aggregated centrally to be useful at all." |
| Staff+ | Separates the consumers and names causality: "logging targets MTTR — we can't stop failing, so we optimize for explaining. But 'logs' is really four pipelines with conflicting needs: debugging wants high detail and short retention, compliance wants immutable low-detail retention for years, breach response can't tolerate sampling, analytics can. And the distributed-specific problem is causality — there's no global clock, so ordering needs a sequencer-issued request ID rather than timestamps." |
Next: what makes this hard once services are distributed.