Initial Design

In one line: this design is right in shape and wrong in scale, and the fix in the next two lessons is entirely about distributing the one component that cannot be centralized.

The components

The standard decomposition is collect, ship, index, visualize — the same four boxes as Elasticsearch/Logstash/Kibana, or Loki and Grafana, or Splunk. Name them immediately and spend your time on the parts that are actually contested.

The accumulator exists to solve Lesson 1's fatal problem

Its stated purpose is exactly the fix for the print-statement failure: "fetch logs from central storage rather than visiting individual nodes."

Lesson 1 established that logs on local stdout die with the node and cannot be correlated across machines. Lesson 2 added that there may be thousands of instances. The accumulator is the component that gets logs off the node — which is the single most important thing this architecture does.

Everything downstream — storage, indexing, visualization — only becomes possible once logs are somewhere central. Notice the ordering: aggregation is a precondition, not a feature.

Blob storage is the right choice, for reasons object storage established

Logs are immutable once written, append-only, written once and read rarely, and enormous in volume. That is the write-once-read-many profile object storage was built around.

The specific properties that pay off: the flat namespace handles billions of log files with no directory bottleneck; immutability means replicas cannot diverge and cached copies cannot be stale; and lifecycle tiering maps directly onto Lesson 8's expiration checker moving old logs to cold storage.

It is also the right cost profile. Logs are read rarely — most log lines are never read by anyone — so paying for hot storage on all of them would be wasteful, and the hot/cool/archive tiers exist for precisely this shape of data.

Indexer plus visualizer is the ELK-style split

The indexer feeds the Distributed Search building block, and the visualizer sits on top of it.

This maps onto tooling you will have seen: an aggregation agent, a search index, and a query/dashboard layer — Elasticsearch, Logstash, and Kibana being the canonical trio, or Loki and Grafana, or Splunk.

Worth recognizing because the interview version of this question is often "design something like Splunk", and knowing the standard decomposition — collect, ship, index, visualize — lets you name the four boxes immediately and then spend your time on the interesting parts.

What do you actually index?

"Index the logs" hides the most expensive decision in the chapter, and an interviewer who has run a log platform will go straight at it.

A full-text index over every log line is not a small addition to storage — it is frequently larger than the logs themselves, because an inverted index stores every term, its postings, and its positions. You are paying to make searchable a corpus in which most lines are never read by anyone.

So there are two positions, and they are a genuine fork:

The label-only design works because of a property this chapter already established: logs are written enormously more often than they are read. Paying full index cost on every line to make a query fast is optimizing the rare operation at the expense of the constant one.

The label set is where it goes wrong. Labels must have low cardinality — service is a good label, user_id is a catastrophic one, because every distinct value creates its own stream and the index explodes back to the size you were avoiding. That is the same failure metrics systems call cardinality explosion, and the same rule applies: labels are for narrowing, fields are for filtering after the scan.

The answer that lands is not picking a side. It is: "index labels by default, and full-text index only the classes of log that are actually searched ad hoc — routing that decision through the filterer, which already knows which application a line belongs to."

Where it breaks

In a distributed system with millions of servers, a single log accumulator creates a scalability bottleneck. We must design the system to scale effectively.

The accumulator is on the write path of every log line in the system — quantify that

Every service on every node writes to it. So its throughput requirement is the sum of the entire fleet's log volume, which is the largest write stream in the system by a wide margin.

Put Lesson 7's figure against it: Meta's logs reach several petabytes per hour. No single component absorbs that.

The failure is worse than slow. If the accumulator saturates, one of two things happens, and both are bad:

• It applies backpressure → applications block on logging → we have violated the "must not block the critical path" requirement, and logging now degrades the product.
• It drops logs → we go blind, and we go blind during incidents, when volume spikes and we most need to see.

There is no acceptable third option, which is why this is a structural flaw rather than a tuning problem.

The fix has two parts, and the next two lessons take one each: make the accumulator per-node rather than global (Lesson 7), and put a horizontally scalable pub-sub tier behind it to absorb the aggregate (Lesson 8).

Notice what the initial design lacks besides scale

Scale is the stated problem. Three other gaps are worth spotting yourself, because the next lessons fill them:

No filtering or routing. All logs go to one store, so multi-tenant separation is impossible and per-application retention policies have nowhere to live.

No alerting. Logs land in storage and wait to be searched. Nobody is told a fatal error occurred — the pipeline is entirely pull, and incidents need push.

No lifecycle. Logs accumulate forever. Lesson 8 adds the expiration checker; without it, storage grows without bound and compliance deletion deadlines cannot be met.

Volunteering these before the interviewer raises them is how you move into the detailed design under your own momentum.

Interview signal by level

Next: fixing it at the node.