Topics, Partitions, and Segments

Why this matters: this is the structure that solves Lesson 4's head-of-line blocking. Immutable records addressed by offset are exactly what let many consumers read independently from one copy.

Key takeaway

A topic is a persistent, immutable sequence of messages. Because disk I/O limits scaling, topics are split into partitions, and a partition consists of messages encapsulated in segments. Segments define the start and end of a message using an offset address, and consumers read from a specific partition offset.

The hierarchy

Level	What it is
Topic	A persistent, immutable sequence of messages. The logical thing producers write to and consumers subscribe to
Partition	A physical division of a topic, stored locally on a broker. Data for a single topic is distributed across multiple partitions
Segment	The unit inside a partition — messages encapsulated in segments, each defining the start and end of a message using an offset address

Why partitions exist

As disk I/O limits scaling, we split topics into partitions.

Partitions are also your unit of parallelism — and therefore your consumer ceiling

Two consequences follow from partitioning that the source states separately but that are worth connecting:

Ordering is per-partition (Lesson 7): strict within, none across.

Consumption is per-partition: a partition is read as one ordered sequence, so within a consumer group one partition maps to at most one consumer.

Put together: your partition count is the maximum number of parallel consumers you can have. Ten partitions means ten consumers maximum; an eleventh sits idle. This is the "ordering domain equals parallelism domain" rule from the Distributed Messaging Queue chapter, made concrete.

Practical implication worth volunteering: over-provision partitions. They are cheap to have and painful to add later, because adding them redistributes keys and breaks any per-key ordering guarantee that consumers were relying on.

Segments and offsets

Messages are stored in segments and identified by an offset. Because records are immutable, readers can access messages independently from any point in the file using the API.

Segment (append-only file)

[offset: 0][offset: 1][offset: 2][offset: 3][offset: 4][offset: 5][offset: 6][offset: 7]
     ^                                                                            ^
 First entry written                                                    Next entry to write

Producer x  ---- writes ---->  appends at the end
Consumer 1  ---- reads  ---->  offset 2
Consumer 2  ---- reads  ---->  offset 6

Both read the same file, at different positions, without blocking each other.

This is exactly the fix for Lesson 4's head-of-line blocking — say it explicitly

Put the two pictures side by side.

In the shared-queue-with-refcount design, consumers could only read from the head. Message m1 could not be removed until the slowest consumer took it, and nothing behind it was reachable. One slow subscriber froze everyone.

Here, consumer 1 sits at offset 2 and consumer 2 at offset 6, reading the same file simultaneously. Consumer 1 being slow, stopped, or offline has zero effect on consumer 2 — there is no head, no removal, and nothing to block on.

The property that makes this work is immutability. Because records are never modified or removed on read, a reader is just a cursor into a file, and cursors do not interfere. Deletion happens later and in bulk, driven by retention.

Immutable log + per-reader offset = independent consumption. That one line is the architectural answer to the whole chapter, and it is what the queue abstraction structurally could not provide.

Why segments rather than one enormous file per partition

The source says a partition is "messages encapsulated in segments" — plural — and the reason is deletion.

Retention requires removing old messages. Deleting a prefix from the middle of one giant file is expensive and fragmenting. But if a partition is a series of segment files, expiry becomes "delete the oldest segment file" — an O(1) filesystem operation with no rewriting and no fragmentation.

Segments also bound recovery and indexing work: after a crash you validate the tail of the active segment rather than scanning terabytes, and an offset-to-position index per segment stays small.

So the three-level hierarchy each earns its place: topics are the logical unit, partitions the parallelism unit, and segments the retention and recovery unit.

Key takeaway

Topic → partitions → segments → offsets. Partitions exist because disk I/O bounds a single broker, and they double as the parallelism ceiling for consumers. Segments make retention deletion cheap. And immutable records addressed by offset are what let many consumers read one copy without blocking each other.

Interview signal by level

Level	What a strong answer sounds like
L4	"Topics are split into partitions across brokers."
L5	Gives the reason and the mechanism: "partitions exist because one broker's disk I/O caps throughput, and within a partition messages sit in append-only segments addressed by offset, so consumers read from wherever they are."
Staff+	Connects it back to the failed design and forward to capacity: "immutability plus per-reader offsets is what fixes head-of-line blocking — a queue only serves from the head, so one slow consumer froze everyone; a log gives every reader an independent cursor. Partition count is also the consumer parallelism ceiling, so I'd over-provision, because adding partitions later redistributes keys and breaks per-key ordering."

Next: which partition a message goes to, and how to force ordering.

Topics, Partitions, and Segments

The hierarchy

Why partitions exist

Segments and offsets

Interview signal by level

Enjoying the preview?