Tracing a Request End to End

In one line: this solves the problem Lesson 2 posed and Lesson 5's requirement demanded. Everything else in this chapter collects logs; this is what makes them readable.

The mechanism

Most complex services use a frontend server to handle an end user's request. Upon receiving a request, the frontend server obtains a unique identifier using a sequencer. This unique identifier is appended to all the fanned-out services. Each log message generated anywhere in the system also emits the unique identifier.

Later, we can filter the log (or preprocess it) based on the unique identifiers. At this step, we can collect all logs across microservices for a given request.

Filter all logs where trace_id = 10472, sort by ID:

10472  frontend   received request
10472  service A  calling B
10472  service B  cache miss
10472  service B  database timeout   <- root cause
10472  service A  B returned 500
10472  frontend   returning 503

This is the answer to Lesson 2's cascade — it tells you what failed first

Lesson 2's problem was that a cascade produces too much evidence: every service in the chain logs an error, and the loudest is the user-facing one that broke last.

Filtering by trace ID collapses that. Instead of a dozen services' full logs, you get one request's story in order — and the first error in that ordered list is the root cause. The frontend's 503 is visibly a consequence, not a cause.

That is the whole value: from "which of these twelve angry services started it" to "read the list top to bottom."

Why a sequencer, not a timestamp

In the Sequencer building block, we discussed how to obtain unique identifiers that maintain happens-before causality. Such an identifier has the property that if ID 1 is less than ID 2, then ID 1 represents a time that occurred before ID 2. Now each log item can use a timestamp, and we can sort log entries for a specific request in ascending order.

Correctly ordering the log in chronological (or causal) order simplifies log analyses.

Wall-clock timestamps across machines cannot order events reliably — that's the whole reason for a sequencer

The natural instinct is to sort by timestamp. It does not work, and Lesson 1 flagged why: there is no global clock.

Machine clocks drift. Even with NTP, skew of several milliseconds between servers is normal, and clocks can jump backwards during correction. So two events milliseconds apart on different machines may be recorded in the wrong order — and in a cascade, the interesting events are exactly that close together.

The failure is silent and actively misleading. You do not get "unknown order"; you get a confidently wrong order, which points at the wrong root cause.

The sequencer solves it by guaranteeing happens-before: ID 1 < ID 2 means event 1 really did occur first, regardless of which machine recorded it. That is unique ID generation's whole purpose — unique IDs that are also globally ordered — reused here.

This is the strongest possible argument for that building block: ordering across machines is not a timestamp problem, it is a sequencing problem.

Propagation is the hard part in practice, and it is easy to break

The mechanism sounds simple: attach an ID, pass it along. Operationally it is the fragile part.

Every service must forward it. One service that drops the header severs the trace — everything downstream of it becomes unattributable, and you get a story that stops halfway with no indication it was truncated.

It must cross every boundary: HTTP headers, RPC metadata, message queue headers, thread-local storage across async boundaries. Each is a place it can be lost, and async handoffs are the classic offender.

It must reach third-party and legacy services, which may not cooperate at all.

This is why distributed tracing is usually implemented as framework middleware rather than application code — the propagation happens automatically at the RPC layer, so an individual developer cannot forget it. And it is why standards like OpenTelemetry and W3C Trace Context exist: a trace only works if everyone agrees on the header name.

The transferable point: the design is trivial and the adoption is not. A tracing system's value is proportional to its coverage, and partial coverage produces traces that are misleading rather than merely incomplete.

What this design leaves out: spans and causality within a request

The design gives every log line in a request the same ID, which answers "what happened during this request, in what order?"

Full distributed tracing goes further with spans — a parent-child tree recording which service called which, and how long each took. That answers different questions: "which service consumed the latency?" and "was B called by A or by C?"

The chapter's flat trace ID plus ordering is the simpler 80% solution, and it is what the design describes. Naming the next step — "I'd extend this to parent-child spans if we needed a latency breakdown per service" — shows you know where this sits on the spectrum without overbuilding what was asked for.

This is also what makes correct sampling possible

Lesson 3 established that sampling independently per service shatters traces — three services at 10% each yields a complete trace 0.1% of the time.

The trace ID is what fixes it. Decide once at the frontend, based on the ID, whether this request is sampled, and propagate that decision alongside the ID. You then keep complete traces for 10% of requests rather than fragments of all of them.

So the trace ID does double duty: it reassembles traces at query time, and it preserves them at sampling time. Both depend on the same propagation.

When is a trace complete?

A question that catches people out, because the answer is that you never actually know.

A trace is assembled from log lines that arrive independently, from different services, over different network paths, through per-node buffers that flush on their own schedules. There is no message announcing "that was the last one." A service may still be running; its accumulator may not have flushed yet; a slow leg may deliver ten seconds late.

So completeness is decided by a timeout, not by a signal: hold the partial trace for a settling window, then treat it as final. The window is the usual trade — too short and you finalize traces that are missing their tail, which is exactly the tail that explains a slow request; too long and the trace is not queryable while someone is trying to use it during an incident.

Two things follow that are worth saying out loud. Anything arriving after the window is late data, and the system needs a policy for it — usually attach it to the stored trace but do not re-run any alerting that already fired on the earlier version. And a partial trace must be labelled partial, because a trace that silently stops looks identical to a request that legitimately ended there, and that ambiguity sends people chasing a service that was never involved.

Who watches the logging system?

The chapter has argued throughout that the log pipeline must work hardest exactly when the application is least healthy. That raises the obvious question, and there is one clearly wrong answer.

Monitoring the logging system with the logging system is circular: when it fails, the evidence of the failure is written into the thing that failed. Teams discover this during an incident, which is the worst possible time to learn it.

The answer is a separate and deliberately simpler meta-monitor on different infrastructure, watching a handful of signals: is the pipeline accepting writes, how far behind are the consumers, how many lines are being dropped, and is anything at all arriving from each data center. It does not need to be sophisticated — it needs to fail independently.

The general principle is worth carrying beyond logging: a diagnostic system must not depend on the system it diagnoses. It is the same reason on-call runbooks are not stored only in the service being debugged.

Conclusion

Logging helps trace event flow in a distributed system and reduces mean time to repair (MTTR) by supporting root cause analysis. Because logging is I/O-intensive, it should be handled asynchronously to avoid blocking the critical request path. Logs feed alerting systems and error aggregation pipelines to monitor application health.

What this design gives up — worth volunteering

Logs can be lost. Buffering in RAM to stay off the critical path means a crashing process loses its unflushed buffer — the logs that explain the crash.
The trace breaks if any service drops the ID, and the resulting partial trace is misleading rather than obviously incomplete.
No spans, so you get ordering but not a per-service latency breakdown.
Pub-sub retention is finite — a consumer down longer than the window loses that data permanently.
Sampling is out of scope here, and adding it later requires the per-request decision, not per-service.
Trace completeness is decided by a timeout, so a trace can be finalized while its slowest leg is still arriving.
The pipeline needs its own separate monitor, or the failure of the logging system is invisible by construction.

Key takeaway

The frontend obtains a sequencer-issued ID and every service propagates it, so filtering by that ID turns a cascade's dozen angry services into one ordered story whose first error is the root cause. Wall-clock timestamps cannot do this — skew produces a confidently wrong order. The design is trivial; propagation coverage is the hard part, and one service dropping the header severs the trace silently. Completeness is decided by a timeout, not a signal, so a partial trace must be labelled partial. And the pipeline needs a separate, simpler monitor — a diagnostic system must not depend on the system it diagnoses.

Interview signal by level

Level	What a strong answer sounds like
L4	"Attach a request ID to every log line so we can filter by it later."
L5	Places the ID's origin: "the frontend generates it and every downstream service propagates it, so filtering by that ID gives the whole request's path across services."
Staff+	Insists on the sequencer and names the operational risk: "it can't be a timestamp — clock skew across machines is milliseconds and clocks jump backwards during NTP correction, so sorting by timestamp gives a confidently wrong order, which points at the wrong root cause. A sequencer gives happens-before, so ID ordering is real ordering. The hard part is propagation: one service dropping the header severs the trace, and async boundaries are where it's usually lost — which is why this belongs in framework middleware, not application code. Same ID also fixes sampling: decide once at the frontend and propagate, so you keep complete traces for 10% of requests instead of fragments of all of them. And two operational points worth volunteering: a trace is finalized on a settling timeout rather than a signal, so partial traces have to be labelled partial — a truncated trace looks identical to a request that legitimately ended, and that ambiguity sends people chasing a service that was never involved. And the pipeline needs its own monitor on separate infrastructure, because monitoring the logging system with the logging system means the evidence of its failure goes into the thing that failed."

Next: the whole design under interview conditions.