Storage: Why Local Disk and Not a Blob Store

In one line: "why not just use S3?" is a question interviewers genuinely ask, and the answer here is one of the few places in system design where spinning-disk physics is the deciding argument.

Why not a blob store?

"Why can't we use blob stores like S3 to keep messages, instead of the broker's local storage?"

Blob stores like S3 are not optimized for writing or reading small amounts of data. If our data is geo-replicated, the above problem is exacerbated.

Therefore, we used the server's local persistent store with append-based writing. Traditional hard disks are specifically tuned to deliver good write performance when writing to contiguous tracks or sectors. Reading throughput and latency is also good for contiguous regions of the disk because it allows extensive data caching.

The real argument is per-operation overhead, and you have seen it twice already

The 1 MB cap from Lesson 3 means messages are small. Small records against a system with high per-operation cost is the worst possible pairing: you pay an HTTP request, TLS, authentication, and a network round trip to move a kilobyte.

That is the same finding as Redis pipelining in distributed caching — 5x throughput even on loopback, because when the work per operation is tiny, the ceremony around it is the system. And it is the same finding as batching in the messaging queue chapter.

An append to a local file skips all of it. The write goes to page cache and returns; the OS flushes contiguously later. That is why the design uses append-based writing specifically — not just local disk, but local disk in the one pattern disks are fastest at.

'Contiguous tracks or sectors' is a real physical claim

On a spinning disk, a random write means a seek — physically moving the head — costing milliseconds. A sequential write costs essentially nothing extra because the head is already there.

The gap is roughly three orders of magnitude, which is why sequential-write designs (append-only logs, LSM trees, write-ahead logs) recur throughout storage systems. The Databases chapter's WAL is the same idea for the same reason.

On SSDs the seek penalty largely disappears, but sequential writes still win: they align with erase-block boundaries, reduce write amplification, and extend device life. The conclusion survives the hardware change even though the mechanism doesn't — which is a good thing to be able to say if an interviewer points out that nobody runs brokers on spinning disks anymore.

This is a considered trade-off, not a free win

Local disk buys speed and costs you things a blob store gives away:

• Storage is bounded by the broker's disk, so retention is capped by hardware. A blob store is effectively unbounded.
• Durability is your problem. S3 replicates for you; here the cluster manager must maintain replicas (Lesson 9).
• Storage and compute are coupled. Adding storage means adding brokers, even if you have plenty of CPU.

Modern systems have started unpicking this — tiered storage that keeps recent segments on local disk and ages older ones into object storage, capturing the write path's speed and the blob store's capacity. Mentioning that shows you know the trade-off is being actively renegotiated as networks get faster, without contradicting the design's reasoning for the hot path.

What makes the log fast

Sequential append explains why writing is cheap. Three further mechanisms explain why a single broker can move far more data than the per-message arithmetic suggests, and they are worth naming because an interviewer probing "how is this so fast?" is asking for exactly these.

Batching is the one that matters most, and it is the same idea as the segment append: the fixed costs here are per-operation, not per-byte, so the way to make them small is to put more bytes in each operation. A producer accumulates messages for a few milliseconds and sends them as one request. That trades a little latency for a large throughput gain, and the size of the trade is a tunable rather than a fixed property.

Compression applied to the batch is what makes it more than a network saving. Messages on one topic tend to be structurally similar — same fields, same shapes — so a batch of them compresses far better than any single one would. And because the broker stores the batch still compressed and hands it to consumers still compressed, one compression on the producer buys reduced network cost on the write, reduced disk cost in storage, and reduced network cost on every read. The CPU is spent once at the edge.

Zero-copy is the mechanism behind the page-cache claim above. The naive read path copies bytes from page cache into the application, then back into the kernel's socket buffer, for no purpose — the broker never looks at message contents. Handing the kernel a file range and a socket instead removes those copies entirely, so a consumer read costs little more than the network transfer itself.

Note what makes all three legal: the broker never interprets the payload. It does not parse, validate, filter, or transform, which is precisely why it can pass compressed batches straight through from disk to socket. The "deliberately dumb broker" from the high-level design is not just an architectural preference — it is the enabling condition for the performance.

Spreading partitions across brokers

"What problems can arise if all partitions are on the same broker?"

If the broker fails or dies, all the messages in the partitions will be lost. To avoid this, we need to ensure the partitions are spread across different brokers.

Partitions are distributed across different brokers in the cluster.

Two views of the same thing, both worth being able to draw:

• Physical view: a broker contains multiple topics, and a topic's partitions are spread across other brokers.
• Logical view: topic B is split into partitions 0, 1, 2 … n, which the consumer reasons about without caring which broker holds each.

Distribution alone is not durability — note what this does and does not fix

Spreading partitions across brokers limits the blast radius: losing one broker loses only the partitions it held, not the whole topic. Good, and necessary.

But those partitions are still gone. Distribution converts "lose everything" into "lose a fraction" — it does not convert it into "lose nothing". That requires replication, which is Lesson 9's job.

Keeping these separate is worth doing in an interview, because candidates routinely say "we partition across brokers so we don't lose data," which is not what partitioning does. Partitioning contains failure; replication survives it.

Making messages findable

"If we use a round robin algorithm to send messages to a partition, how does the system know where to look when it is time to read?"

The system will need to persist appropriate metadata. This metadata will keep mappings between the logical index of segment or messages to the server identity or partition identifier.

The consumer manager (Lesson 10) holds this information.

Round robin creates a lookup problem that keyed placement does not

This question exists precisely because of round robin. If a producer supplies a partition_ID, or if the system hashes a key, then finding a message is deterministic — recompute the placement and you know where it went.

Round robin has no such rule. Placement depends on which partition happened to be next when the message arrived, so nothing can be recomputed. The mapping has to be remembered, which is why the design needs a metadata layer mapping logical index → server identity or partition identifier.

This is the classic trade the Databases and Distributed Cache chapters both made: computed placement (consistent hashing) needs no lookup but constrains distribution; directory-based placement allows arbitrary distribution but requires a lookup and a store to keep it in. Round robin is directory-based, which is why the metadata exists at all.

Interview signal by level

Next: replication and what happens when a broker dies.