Storage: Why Local Disk and Not a Blob Store

Why this matters: "why not just use S3?" is a question interviewers genuinely ask, and the answer here is one of the few places in system design where spinning-disk physics is the deciding argument.

Key takeaway

Blob stores like S3 are not optimized for writing or reading small amounts of data, and geo-replication exacerbates the problem. The design instead uses the server's local persistent store with append-based writing.

Why not a blob store?

"Why can't we use blob stores like S3 to keep messages, instead of the broker's local storage?"

Blob stores like S3 are not optimized for writing or reading small amounts of data. If our data is geo-replicated, the above problem is exacerbated.

Therefore, we used the server's local persistent store with append-based writing. Traditional hard disks are specifically tuned to deliver good write performance when writing to contiguous tracks or sectors. Reading throughput and latency is also good for contiguous regions of the disk because it allows extensive data caching.

	Blob store (S3)	Local disk, append-only
Optimized for	Large objects, infrequently accessed	Small records, written and read constantly
Per-operation cost	An HTTP round trip per object — dominates when the object is 1 KB	An append to an open file, often just a page-cache write
Access pattern	Random object addressing	Contiguous tracks and sectors — the pattern disks are tuned for
Caching	Remote; limited benefit	Extensive — contiguous regions cache well
Geo-replication	Makes it worse — every small operation crosses regions	Replication is broker-to-broker, batched

The real argument is per-operation overhead, and you have seen it twice already

The 1 MB cap from Lesson 3 means messages are small. Small records against a system with high per-operation cost is the worst possible pairing: you pay an HTTP request, TLS, authentication, and a network round trip to move a kilobyte.

That is the same finding as Redis pipelining in the Distributed Cache chapter — 5x throughput even on loopback, because when the work per operation is tiny, the ceremony around it is the system. And it is the same finding as batching in the messaging queue chapter.

An append to a local file skips all of it. The write goes to page cache and returns; the OS flushes contiguously later. That is why the design uses append-based writing specifically — not just local disk, but local disk in the one pattern disks are fastest at.

'Contiguous tracks or sectors' is a real physical claim

On a spinning disk, a random write means a seek — physically moving the head — costing milliseconds. A sequential write costs essentially nothing extra because the head is already there.

The gap is roughly three orders of magnitude, which is why sequential-write designs (append-only logs, LSM trees, write-ahead logs) recur throughout storage systems. The Databases chapter's WAL is the same idea for the same reason.

On SSDs the seek penalty largely disappears, but sequential writes still win: they align with erase-block boundaries, reduce write amplification, and extend device life. The conclusion survives the hardware change even though the mechanism doesn't — which is a good thing to be able to say if an interviewer points out that nobody runs brokers on spinning disks anymore.

This is a considered trade-off, not a free win

Local disk buys speed and costs you things a blob store gives away:

Storage is bounded by the broker's disk, so retention is capped by hardware. A blob store is effectively unbounded.
Durability is your problem. S3 replicates for you; here the cluster manager must maintain replicas (Lesson 9).
Storage and compute are coupled. Adding storage means adding brokers, even if you have plenty of CPU.

Modern systems have started unpicking this — tiered storage that keeps recent segments on local disk and ages older ones into object storage, capturing the write path's speed and the blob store's capacity. Mentioning that shows you know the trade-off is being actively renegotiated as networks get faster, without contradicting the design's reasoning for the hot path.

Spreading partitions across brokers

"What problems can arise if all partitions are on the same broker?"

If the broker fails or dies, all the messages in the partitions will be lost. To avoid this, we need to ensure the partitions are spread across different brokers.

Partitions are distributed across different brokers in the cluster.

Two views of the same thing, both worth being able to draw:

Physical view: a broker contains multiple topics, and a topic's partitions are spread across other brokers.
Logical view: topic B is split into partitions 0, 1, 2 … n, which the consumer reasons about without caring which broker holds each.

Making messages findable

"If we use a round robin algorithm to send messages to a partition, how does the system know where to look when it is time to read?"

The system will need to persist appropriate metadata. This metadata will keep mappings between the logical index of segment or messages to the server identity or partition identifier.

The consumer manager (Lesson 10) holds this information.

Round robin creates a lookup problem that keyed placement does not

This question exists precisely because of round robin. If a producer supplies a partition_ID, or if the system hashes a key, then finding a message is deterministic — recompute the placement and you know where it went.

Round robin has no such rule. Placement depends on which partition happened to be next when the message arrived, so nothing can be recomputed. The mapping has to be remembered, which is why the design needs a metadata layer mapping logical index → server identity or partition identifier.

This is the classic trade the Databases and Distributed Cache chapters both made: computed placement (consistent hashing) needs no lookup but constrains distribution; directory-based placement allows arbitrary distribution but requires a lookup and a store to keep it in. Round robin is directory-based, which is why the metadata exists at all.

Key takeaway

Small messages plus high per-operation cost is why blob stores lose here; append-only writes to contiguous disk is the one pattern disks are fastest at. Spreading partitions across brokers contains failure but does not survive it. And round-robin placement forces a metadata mapping from logical index to physical location.

Interview signal by level

Level	What a strong answer sounds like
L4	"Messages are stored on the broker's disk."
L5	Explains the pattern: "append-only writes to local disk, because sequential writes to contiguous regions are far faster than random ones, and blob stores aren't built for constant small reads and writes."
Staff+	Generalizes and states the cost: "messages are capped at 1 MB, so per-operation overhead dominates — an HTTP round trip to move a kilobyte is the same mistake as not pipelining Redis. The trade is that storage is bounded by broker disk and durability becomes ours, which is why tiered storage into object storage for cold segments is where these systems are heading. And spreading partitions contains a broker failure; it doesn't survive one — that's replication's job."

Next: replication and what happens when a broker dies.

Storage: Why Local Disk and Not a Blob Store

Why not a blob store?

Spreading partitions across brokers

Making messages findable

Interview signal by level

Enjoying the preview?