Partition Assignment and Strict Ordering

In one line: the ordering mechanism here is a client-supplied partition ID, which looks like leaking implementation detail into the API. It is actually the cheapest correct answer, and being able to defend that is a strong interview moment.

Default placement: weighted round robin

Messages cycle across partitions in turn, spreading write load evenly.

Why weighted, and not plain, round robin

Plain round robin assumes every partition can absorb the same rate. In a real cluster that is false: brokers differ in disk speed, in how many other partitions they host, and in how much traffic those partitions carry.

Weighting lets each partition receive a share proportional to its capacity, so a partition on a fast, lightly-loaded broker takes more and a busy one takes less.

This is the same refinement load balancing made — round robin to weighted round robin — and the same reasoning behind virtual nodes sized to machine capability in distributed caching. Uniform distribution is only correct when the targets are uniform, and at fleet scale they never are.

Strict ordering via partition ID

"Strict ordering ensures that messages are stored in the order in which they are produced. How can we ensure strict message ordering?"

We'll assign each partition a unique ID, partition_ID. The user can provide the partition_ID while writing into the system. In this way, the messages will be sent to the specified partition, and the ordering will be strict.

The write call becomes:

write(topic_ID, partition_ID, message)

If the user does not provide the partition_ID, we'll use a weighted round robin algorithm to determine which message to send to which partition.

Ordering is opt-in per message, and the default gives you none

This is worth stating plainly because it surprises people. With round-robin placement, messages A, B, C from the same producer land on three different partitions — and since partitions are independent logs read by independent consumers, no ordering relationship survives at all.

That is not a bug; it is the price of the parallelism that makes the system fast. But it means the default behaviour provides no ordering whatsoever, and any correctness that depends on order must explicitly ask for it.

The practical pattern: derive partition_ID from a key whose ordering matters. In the chat application from Lesson 2, partition by channel ID — every message in a channel lands on one partition and is strictly ordered, while different channels spread across partitions and stay parallel.

Choose your partition key to be the smallest thing that needs ordering. That is the same "ordering domain equals parallelism domain" rule, now with a concrete knob.

Client-chosen partitions have a real cost: hot partitions

Every one of these buys load balance by giving up some ordering or adding consumer-side work. There is no option that keeps both.

Handing the client the partition ID means the client can choose badly, and the system has no defence.

Round robin guarantees even distribution. A client-supplied key does not: partition by country and one partition gets most of the traffic; partition by customer and your largest customer saturates a broker while the rest idle. The weighted round robin that was carefully balancing load is bypassed entirely for keyed writes.

This is the hot-shard problem from distributed caching, arriving through the API. Same mitigations: pick a high-cardinality, evenly-distributed key, or split a hot key artificially (channel:1, channel:2) and accept losing order across the splits.

Worth saying out loud in an interview: "partitioning by key trades guaranteed load balance for guaranteed ordering, and the risk is a hot partition."

Why let the client choose at all?

It might seem strange to give the client of pub-sub the ability to choose a partition. However, such a facility can serve as a basis for clients to obtain data for a specific time period — for example, data from yesterday. For simplicity, we won't include time-based reading in our design.

The source flags its own oddity — and the answer is about who has the information

Exposing partitions looks like a leaked implementation detail, and normally it would be. Here it is the right call for a specific reason: only the client knows which messages are related.

The system sees an opaque payload. It cannot know that these three messages belong to the same chat channel, the same user session, or the same database row. The semantic relationship that ordering is about exists only in the client's head.

This is the same conclusion message-queue design reached about sequence numbers — "only the producer knows production order." The system could invent an ordering, but it would be ordering by arrival, which is not the thing anyone wanted.

So the API exposes the minimum needed to let the client express "these belong together". Kafka does exactly this, with one refinement worth mentioning: it takes a message key and hashes it to a partition, rather than taking a raw partition number. Same guarantee, one less implementation detail leaked — the client says what is related, not where to put it.

The hint about time-based reads is real, and worth understanding

"A basis for clients to obtain data for a specific time period — for example, data from yesterday" points at something the design leaves out but that real systems build.

If you can address a partition and an offset directly, then given an index from timestamp to offset you can seek to "the first message after midnight yesterday" and read forward. That turns the log into a time-queryable archive rather than just a delivery mechanism.

The design excludes it for simplicity, but it explains why the addressing model is exposed at all — and it is the mechanism behind Kafka's offsetsForTimes. If an interviewer asks how to replay a specific window, this is the answer: timestamp index → offset → read forward.

Interview signal by level

Next: why the messages sit on local disk.