Consumer Manager: Offsets, Retention, and Delivery

Why this matters: offsets are the mechanism the entire chapter has been building toward. One log, many cursors is what makes independent consumption possible, and this is the component that holds the cursors.

Key takeaway

The consumer manager handles four things: verifying the consumer, retention time management, message delivery options (push or pull), and offset management in a key-value store.

Responsibility	Detail
Verify the consumer	Checks if a consumer is authorized to read from a specific topic
Retention time management	Enforces retention policies to prevent access to expired messages
Message delivery options	Supports both push (system sends data) and pull (consumer requests data). This flexibility prevents consumer overload while supporting real-time needs. Preferences are stored in the database
Offset management	Stores each consumer's offset in a key-value store, allowing consumers to resume reading from where they left off

Offset management

Partition 0 of Topic A  (immutable, append-only)

[ off 0 ][ off 1 ][ off 2 ][ off 3 ][ off 4 ][ off 5 ][ off 6 ][ off 7 ]
              ^                                  ^
        Consumer B: 2                      Consumer A: 6

Key-value store
  (consumer_B, topicA:p0) -> 2
  (consumer_A, topicA:p0) -> 6

One copy of the data. One small mutable value per consumer per partition.

Why a key-value store and not the relational database

Note the deliberate split. Subscriptions, consumer metadata, and retention policies live in the relational store (Lesson 5). Offsets live in a key-value store. Both are "metadata," and they are separated on purpose.

Their access patterns are nothing alike:

Subscriptions: low volume, changed rarely, queried relationally — "which consumers subscribe to topic X" is a join, and integrity matters.
Offsets: enormous volume — one per consumer per partition — overwritten constantly as consumers advance, and only ever fetched by exact key. No joins, no relations, no transactions across them.

A million consumers across ten partitions each is ten million offsets updating continuously. That is a workload a relational database handles badly and a key-value store handles by design, which is precisely the profile the Key-Value Store chapter described.

Splitting metadata by access pattern rather than putting it in one store is a choice interviewers notice, and this is a clean example to point at.

When the offset is committed decides your delivery semantics

This is the highest-value detail in the lesson, and the source leaves it implicit.

A consumer reads a message, processes it, and advances its offset. The order of the last two steps determines the guarantee:

Commit the offset after processing → if the consumer crashes between processing and committing, it restarts at the old offset and reprocesses. That is at-least-once, and it requires idempotent handlers — the same conclusion the Distributed Messaging Queue chapter reached.
Commit the offset before processing → if it crashes mid-processing, the offset has already moved and the message is never reprocessed. That is at-most-once, and it means dropped work.

There is no ordering of two independent operations that gives exactly-once. Getting it requires either a transaction spanning the offset commit and the side effect, or an idempotent handler that makes duplicates harmless.

If an interviewer asks how you get exactly-once from this design, the honest answer is: you don't get it from the delivery mechanism — you get it by making processing idempotent, or by making the offset commit and the side effect atomic.

Push or pull, per consumer

Supports both models, and the reason given is precise: this flexibility prevents consumer overload while supporting real-time needs. Preferences are stored in the database.

Model	Fits	Risk
Pull	Batch consumers, analytics, anything with variable processing speed	Poll latency — mitigated by long polling
Push	Real-time needs — chat clients, live dashboards	Consumer overload, since the consumer no longer controls the rate

Per-consumer choice is the right resolution of the pull-versus-push debate

The Distributed Messaging Queue chapter argued strongly for pull, on backpressure grounds: under push, an overload relocates the backlog into the consumer, which has no durability.

That argument still holds — and it is why preference is per consumer rather than system-wide. Different subscribers to the same topic have genuinely different needs. A chat client wants push, because a poll interval is visible to a human. A nightly analytics job wants pull, because it processes in large batches at its own pace.

Storing the preference in the database, per consumer, lets both share one topic without compromise. That is a nicer answer than picking a side, and it is only possible because the log is the source of truth: push and pull are both just ways of moving a cursor forward over the same immutable data.

Retention enforcement

Enforces retention policies to prevent access to expired messages.

A consumer slower than retention hits a gap, silently

Retention expiry is time-driven and does not consult anyone's offset. So a consumer whose offset points into an already-expired region is asking for data that no longer exists.

The failure is quiet unless you look for it. The consumer's position is simply before the start of the log, and its next read either errors or jumps forward to the oldest surviving message — losing everything in between with no signal that anything was skipped.

Two defences:

Alert on consumer lag against the retention window, not just against the head. Lag measured in messages tells you a consumer is behind; lag measured against retention tells you it is about to lose data.
Treat "offset before the log start" as a hard error requiring a decision, not something to silently paper over by seeking to the earliest offset.

This is the operational hazard flagged back in Lesson 2's durability discussion, now with the mechanism attached.

Key takeaway

Offsets in a key-value store — chosen over the relational store because they are high-volume, constantly overwritten, and fetched by exact key — give every consumer an independent cursor over one copy of the log. When you commit the offset decides at-least-once versus at-most-once. Push or pull is a per-consumer preference. And a consumer slower than retention loses data quietly.

Interview signal by level

Level	What a strong answer sounds like
L4	"We store where each consumer has read so it can resume."
L5	Justifies the store: "offsets go in a key-value store — one per consumer per partition, constantly overwritten, only ever fetched by key, so a relational database is the wrong shape."
Staff+	Ties commit order to semantics: "committing the offset after processing gives at-least-once and needs idempotent handlers; committing before gives at-most-once and drops work. There's no ordering that gives exactly-once — you get it from idempotency or from making the commit and the side effect atomic. And I'd alert on lag against the retention window, because a consumer that falls behind retention loses messages with no error."

Next: the finished picture.

Consumer Manager: Offsets, Retention, and Delivery

Offset management

Push or pull, per consumer

Retention enforcement

Interview signal by level

Enjoying the preview?