Topics, Partitions, and Segments

In one line: this is the structure that solves Lesson 4's head-of-line blocking. Immutable records addressed by offset are exactly what let many consumers read independently from one copy.

The hierarchy

Why partitions exist

As disk I/O limits scaling, we split topics into partitions.

Note the stated reason — it is narrower and better than 'to scale'

The justification is specifically disk I/O, not generic scalability, and the distinction matters.

A topic that lived on one broker would be bounded by that machine's disk throughput — a hard physical ceiling that no amount of caching or better code removes. Partitioning is how you get more spindles working on one logical topic.

That framing also tells you how to size partitions: enough that aggregate disk throughput exceeds peak write rate, with headroom. It is a capacity calculation, not a guess — the same back-of-the-envelope reasoning the Foundations module set up.

And it explains why partition count is hard to change later. Repartitioning means moving data and, as Lesson 7 shows, breaking ordering assumptions.

Partitions are also your unit of parallelism — and therefore your consumer ceiling

Two consequences follow from partitioning that the design states separately but that are worth connecting:

Ordering is per-partition (Lesson 7): strict within, none across.

Consumption is per-partition: a partition is read as one ordered sequence, so within a consumer group one partition maps to at most one consumer.

Put together: your partition count is the maximum number of parallel consumers you can have. Ten partitions means ten consumers maximum; an eleventh sits idle. This is the "ordering domain equals parallelism domain" rule from message-queue design, made concrete.

Practical implication worth volunteering: over-provision partitions. They are cheap to have and painful to add later, because adding them redistributes keys and breaks any per-key ordering guarantee that consumers were relying on.

Consumer groups: one system, both delivery models

Everything so far describes a log that any number of readers can scan independently. That gives pub-sub semantics. The mechanism that also gives you queue semantics — without a second system — is the consumer group.

A consumer group is a named set of consumers that cooperate. The rule is one line: each partition is assigned to exactly one consumer within a group.

Read it in two directions and you get both models out of one structure:

• Within a group, the partitions are divided up, so each message is handled by exactly one member. Adding a consumer splits the work. That is a queue.
• Across groups, each group holds its own independent set of offsets, so every group sees every message. Adding a group duplicates the work. That is pub-sub.

So the distinction from the first lesson — queue splits, pub-sub duplicates — is not a choice between two systems. It is a choice about whether two consumers share a group name, and that is a remarkably small lever for so large a semantic difference.

It also explains why offsets are stored per group and per partition rather than per consumer. A consumer is transient — it crashes, restarts, gets replaced by a different instance. The group's position in the log has to outlive any particular member.

Rebalancing is the cost, and it is not free

Because the assignment is dynamic, it must be recomputed whenever the membership changes — a consumer joins, crashes, or is deployed.

Two consequences worth volunteering. A rolling deploy of a ten-consumer group triggers a rebalance per instance, so a routine deploy can mean ten pauses — which is why real systems care a great deal about making rebalances short. And because a moved partition resumes from the last committed offset, rebalancing is a duplicate-delivery event, not merely a pause: work done since the last commit is redone by whoever picks the partition up.

That is the same at-least-once property the offset-commit discussion arrives at later, reached from a different direction, and it is why consumers should be idempotent rather than merely fast.

Segments and offsets

Messages are stored in segments and identified by an offset. Because records are immutable, readers can access messages independently from any point in the file using the API.

Segment (append-only file)

[offset: 0][offset: 1][offset: 2][offset: 3][offset: 4][offset: 5][offset: 6][offset: 7]
     ^                                                                            ^
 First entry written                                                    Next entry to write

Producer x  ---- writes ---->  appends at the end
Consumer 1  ---- reads  ---->  offset 2
Consumer 2  ---- reads  ---->  offset 6

Both read the same file, at different positions, without blocking each other.

This is exactly the fix for Lesson 4's head-of-line blocking — say it explicitly

Put the two pictures side by side.

In the shared-queue-with-refcount design, consumers could only read from the head. Message m1 could not be removed until the slowest consumer took it, and nothing behind it was reachable. One slow subscriber froze everyone.

Here, consumer 1 sits at offset 2 and consumer 2 at offset 6, reading the same file simultaneously. Consumer 1 being slow, stopped, or offline has zero effect on consumer 2 — there is no head, no removal, and nothing to block on.

The property that makes this work is immutability. Because records are never modified or removed on read, a reader is just a cursor into a file, and cursors do not interfere. Deletion happens later and in bulk, driven by retention.

Immutable log + per-reader offset = independent consumption. That one line is the architectural answer to the whole chapter, and it is what the queue abstraction structurally could not provide.

Why segments rather than one enormous file per partition

Retention requires removing old messages. Deleting a prefix from the middle of one giant file is expensive and fragmenting. But if a partition is a series of segment files, expiry becomes "delete the oldest segment file" — an O(1) filesystem operation with no rewriting and no fragmentation.

Segments also bound recovery and indexing work: after a crash you validate the tail of the active segment rather than scanning terabytes, and an offset-to-position index per segment stays small.

So the three-level hierarchy each earns its place: topics are the logical unit, partitions the parallelism unit, and segments the retention and recovery unit.

Interview signal by level

Next: which partition a message goes to, and how to force ordering.