Observability

In one line: the sampling decision in this lesson is the whole reason distributed tracing is viable, and the design states it in half a sentence.

Managing thousands of servers requires robust observability tools.

Zipkin samples rather than tracing everything, and that is not a compromise

Twitter uses Zipkin, a distributed tracing system, to track request paths and latency across microservices. To reduce overhead, Zipkin samples a percentage of requests rather than tracing every event. Data is aggregated via the Scribe server and stored in key-value stores.

Do the arithmetic on the alternative. Lesson 3 established 289,000 reads per second, and a single timeline request touches many services. Tracing every request would mean:

289,000 requests/second x ~10 spans each = ~3 MILLION spans/second

Each span carries a trace ID, span ID, parent ID, service name, timestamps, and annotations — perhaps 200 bytes. That is 600 MB per second of trace data, before storage or query cost. And the instrumentation itself adds latency to every traced request.

You cannot trace everything at this scale. So Zipkin samples — perhaps 0.1% or 1% of requests.

The crucial point is that sampling does not degrade what tracing is for. Tracing answers questions like "which service is adding latency to this request path?" and "what is the call graph in production?" Those are statistical questions, and a random sample of 1% of three million spans per second is still 30,000 spans per second — an enormous amount of evidence.

What sampling does cost is the ability to answer "what happened to this specific request?" — which is exactly what you want during an incident, and exactly when the traced sample probably missed it.

The standard resolutions are worth naming:

Head-based sampling — decide at the entry point, propagate the decision. Simple, and the decision is made before you know whether the request is interesting.

Tail-based sampling — buffer spans and decide after the request completes, keeping the slow ones and the failures. Far more useful, considerably more expensive.

Always sample errors and slow requests. A hybrid: sample 0.1% of normal traffic and 100% of anything that failed or exceeded a latency threshold.

Sample the ordinary, keep all of the exceptional. The interesting requests are rare, so retaining them costs little — and they are the only ones anyone looks at.

Dynamic configuration is availability infrastructure, not convenience

A library backed by ZooKeeper updates service configurations in real time without requiring restarts.

Easy to read as developer convenience. It is not — it is what makes incident response possible.

Consider the levers you need during an outage: disable a failing feature, change a rate limit, shift traffic away from a bad region, adjust a timeout, turn off an expensive code path. If each requires a deploy and restart across thousands of instances, your mean time to recovery is measured in the length of a deploy cycle.

With dynamic configuration, those are seconds. And more importantly, restarting is itself dangerous during an incident: rolling thousands of instances while the system is already degraded drops connections, empties caches, and can turn a partial outage into a full one.

Configuration you can change without restarting is what turns a deploy-length recovery into a seconds-length one. It is also the mechanism behind feature flags and gradual rollouts, which is how you limit the blast radius of a change in the first place.

ZooKeeper is the right backing store because config changes need strong consistency and change notification — every instance must converge on the same value, quickly, and be told when it changes. That is precisely what a coordination service provides and what a cache does not.

ZooKeeper stores coordination metadata, not application data — and the boundary matters

Many distributed systems use ZooKeeper to manage coordination metadata rather than application data. It provides coordination primitives such as distributed locking and leader election. At Twitter, ZooKeeper has been used to store service registries, Manhattan cluster topology, and related system metadata.

The distinction is worth stating sharply, because misusing ZooKeeper is a common and expensive mistake.

ZooKeeper achieves strong consistency by writing every update through a consensus protocol, which is why it is unsuitable for volume — every write is agreed by a quorum. That is affordable for "which node is the leader" and ruinous for "here is a tweet."

Note what it holds here: the service registry, which Lesson 11 needs for client-side load balancing, and Manhattan cluster topology — which node owns which key range. Both are read by everyone, changed rarely, and must be immediately correct when they do change.

That is the same role the key-value store played in that building block and ZooKeeper played in Uber and Quora. Every partitioned system needs one small, strongly-consistent place that says where things live — and it is always separate from the store that holds the things.

Splunk replacing Loglens, and why logging is separate from tracing

The design notes Splunk "replaces the legacy Loglens service" — another reversal in a chapter full of them.

Worth distinguishing the three observability signals, because they answer different questions and are often conflated:

They differ mainly in cardinality and aggregation. Metrics are pre-aggregated numbers, cheap to store and query, and are what alerts fire on. Logs are unaggregated text, expensive to store and search, and are what you read after an alert. Traces are structured causal chains, sampled because they are the most expensive per request.

Alert on metrics, diagnose with traces, confirm with logs. A system that alerts on logs will be too slow and too noisy; one that has only metrics can tell you something is wrong but never why.

Cardinality, not volume, is what kills a metrics system. A counter labelled by user id is one series per user. The discipline is bounding label values at the point of instrumentation, because nothing downstream can undo an unbounded dimension.

Next: what happens when one tweet gets a million likes at once.