Consumer Manager: Offsets, Retention, and Delivery

In one line: offsets are the mechanism the entire chapter has been building toward. One log, many cursors is what makes independent consumption possible, and this is the component that holds the cursors.

Offset management

Partition 0 of Topic A  (immutable, append-only)

[ off 0 ][ off 1 ][ off 2 ][ off 3 ][ off 4 ][ off 5 ][ off 6 ][ off 7 ]
              ^                                  ^
        Consumer B: 2                      Consumer A: 6

Key-value store
  (consumer_B, topicA:p0) -> 2
  (consumer_A, topicA:p0) -> 6

One copy of the data. One small mutable value per consumer per partition.

Why a key-value store and not the relational database

Note the deliberate split. Subscriptions, consumer metadata, and retention policies live in the relational store (Lesson 5). Offsets live in a key-value store. Both are "metadata," and they are separated on purpose.

Their access patterns are nothing alike:

• Subscriptions: low volume, changed rarely, queried relationally — "which consumers subscribe to topic X" is a join, and integrity matters.
• Offsets: enormous volume — one per consumer per partition — overwritten constantly as consumers advance, and only ever fetched by exact key. No joins, no relations, no transactions across them.

A million consumers across ten partitions each is ten million offsets updating continuously. That is a workload a relational database handles badly and a key-value store handles by design, which is precisely the profile key-value stores described.

Splitting metadata by access pattern rather than putting it in one store is a choice interviewers notice, and this is a clean example to point at.

When the offset is committed decides your delivery semantics

This is the highest-value detail in the lesson, and the design leaves it implicit.

A consumer reads a message, processes it, and advances its offset. The order of the last two steps determines the guarantee:

• Commit the offset after processing → if the consumer crashes between processing and committing, it restarts at the old offset and reprocesses. That is at-least-once, and it requires idempotent handlers — the same conclusion message-queue design reached.
• Commit the offset before processing → if it crashes mid-processing, the offset has already moved and the message is never reprocessed. That is at-most-once, and it means dropped work.

There is no ordering of two independent operations that gives exactly-once. Getting it requires either a transaction spanning the offset commit and the side effect, or an idempotent handler that makes duplicates harmless.

If an interviewer asks how you get exactly-once from this design, the honest answer is: you don't get it from the delivery mechanism — you get it by making processing idempotent, or by making the offset commit and the side effect atomic.

Offsets are also what make consumer groups and replay work

Two capabilities fall out of storing position per consumer, and neither needs new machinery:

Independent progress. Consumer A at offset 6 and consumer B at offset 2 read the same partition without interfering — the fix for Lesson 4's head-of-line blocking, now visible as data.

Replay. An offset is just a number in a mutable store. Set it backwards and the consumer reprocesses history — rebuild a downstream database after a bug, backfill a newly deployed service, re-run a corrected transformation.

The bound is retention. You can only rewind as far as the messages still exist, which is why Lesson 5 called retention the recovery window. A bug found on day eight against a seven-day retention cannot be repaired by replay.

Push or pull, per consumer

Supports both models, and the reason given is precise: this flexibility prevents consumer overload while supporting real-time needs. Preferences are stored in the database.

Per-consumer choice is the right resolution of the pull-versus-push debate

The Distributed Messaging Queue chapter argued strongly for pull, on backpressure grounds: under push, an overload relocates the backlog into the consumer, which has no durability.

That argument still holds — and it is why preference is per consumer rather than system-wide. Different subscribers to the same topic have genuinely different needs. A chat client wants push, because a poll interval is visible to a human. A nightly analytics job wants pull, because it processes in large batches at its own pace.

Storing the preference in the database, per consumer, lets both share one topic without compromise. That is a nicer answer than picking a side, and it is only possible because the log is the design of truth: push and pull are both just ways of moving a cursor forward over the same immutable data.

When a consumer cannot process a message

Everything above assumes processing succeeds or the consumer crashes. The awkward middle case is a message the consumer receives fine and cannot handle — malformed payload, a referenced record that no longer exists, a downstream service returning a permanent error.

This matters more here than in a queue, and the reason is structural. A queue can redeliver a failed message to a different consumer and eventually move it aside, because the queue owns per-message state. A partition is an ordered log read by offset, and there is no way to skip one message without advancing past it:

All three are bad, and the third is the classic poison pill: one unprocessable message takes a consumer down permanently, and restarting does not help because it reads the same offset again.

The pattern that resolves it moves the message out of the ordered log rather than trying to handle it in place:

The key move is that the main consumer commits the offset and keeps going — the message is not lost, because it has been durably written somewhere else first. Ordering for that one message is sacrificed deliberately, in exchange for the partition continuing to flow.

Three things worth saying about this:

Retries belong on a separate topic, not in a loop. Retrying in place holds the partition, so a slow-failing downstream turns one bad message into a stalled consumer. A retry topic with a delay lets the main partition proceed at full speed.

A dead-letter topic is a queue you must actually watch. Messages arriving there are, by definition, ones the system could not handle — so an unmonitored DLQ is a silent data-loss channel with extra steps. It needs an alert on depth.

Not every system gives you this for free. Managed queues like SQS have built-in retry counts and dead-letter routing; a log-based system typically does not, and you build it out of ordinary topics. Knowing which you are working with is a real design input, and choosing the simpler system when you need per-message retry semantics is a legitimate answer.

Retention enforcement

Enforces retention policies to prevent access to expired messages.

Time is not the only retention policy

Deleting by age is right for event streams, where a message describes something that happened and stops mattering. It is wrong for the use case the earlier lesson raised: pub-sub as a replication log, where each message is a state change and a subscriber needs the current state.

Under time-based retention, a replica that has been offline longer than the window cannot be rebuilt — the early history it needs has expired. Under infinite retention, the log grows without bound.

Log compaction is the third option: keep the latest message per key, forever, and discard superseded ones.

The property that makes this work is that the log stays a complete description of current state even though most of its history is gone. Replay it from the beginning and you converge on the right answer, which is exactly what rebuilding a cache or a replica needs.

The trade is that you lose history — you can no longer ask what a key's value was last Tuesday — and it only applies to keyed messages. So the rule of thumb: compact when a message is a fact about a thing; expire by time when a message is a record of an event.

A consumer slower than retention hits a gap, silently

Retention expiry is time-driven and does not consult anyone's offset. So a consumer whose offset points into an already-expired region is asking for data that no longer exists.

The failure is quiet unless you look for it. The consumer's position is simply before the start of the log, and its next read either errors or jumps forward to the oldest surviving message — losing everything in between with no signal that anything was skipped.

Two defences:

• Alert on consumer lag against the retention window, not just against the head. Lag measured in messages tells you a consumer is behind; lag measured against retention tells you it is about to lose data.
• Treat "offset before the log start" as a hard error requiring a decision, not something to silently paper over by seeking to the earliest offset.

This is the operational hazard flagged back in Lesson 2's durability discussion, now with the mechanism attached.

Interview signal by level

Next: the finished picture.