Evaluation: Functional Requirements

In one line: message deletion is the most interesting operation in the whole API, because the two available mechanisms produce entirely different systems from the same requirement.

Queue creation and deletion

Send and receive

Producers deliver messages to specific queues. At the backend, incoming messages are sorted by timestamps to preserve order and placed in the queue. Consumers then retrieve messages from these queues.

When the frontend receives a message, it identifies the primary host or cluster where the queue resides — depending on the replication model — and forwards the request to that entity for enqueueing.

Message deletion: two approaches

1. No immediate deletion (offset tracking)

Messages are not deleted upon consumption. Instead, the system maintains message order, and consumers track their own progress (offsets). A background job deletes messages once expiration conditions are met. Apache Kafka uses this model to allow multiple processes to consume the same message stream.

Queue (append-only log)

[ m1 ][ m2 ][ m3 ][ m4 ][ m5 ][ m6 ][ m7 ]
              ^                 ^
        consumer B        consumer A
         offset 3          offset 6

Both read the same messages. Neither removes anything.
A background job deletes older messages once they expire.

Offset tracking is what makes replay possible

The consequence worth naming: because messages are not destroyed on read, a consumer can rewind its offset and reprocess history.

That single property is why Kafka is used for things a queue traditionally is not — rebuilding a downstream database after a bug, backfilling a new service with past events, running a new consumer group over data that already flowed. The queue becomes a durable log you can read repeatedly rather than a channel that empties as you drain it.

It also makes fan-out trivial. Adding a sixth consumer group costs nothing, because nobody's reads interfere with anyone else's — the pub/sub pattern from Lesson 1, implemented by simply not deleting anything.

The cost is storage: you keep messages for the full retention period whether or not everyone has consumed them, and retention becomes a capacity decision rather than a consumption one.

2. Visibility timeout

Messages are not deleted immediately but are made invisible to other consumers for a specific duration (visibility_timeout). The consumer must explicitly delete the message via an API call after processing. If the consumer fails to delete it within the timeout, the message becomes visible again.

The consumer is responsible for the final deletion. This ensures high durability: if a consumer crashes before processing is complete, the message reappears in the queue for another worker to handle.

This mechanism provides at-least-once delivery semantics. If a worker fails, the message eventually becomes visible again, ensuring it is processed at least once.

What if the timeout expires while the consumer is still working?

Choosing the timeout is the same trade as any liveness interval: too long and a crashed consumer's message sits invisible for that long; too short and healthy slow work gets duplicated. A heartbeat that extends the lease while progress is being made is what lets you set it short without punishing legitimate long jobs.

The message becomes visible, and another worker can receive it — duplicating the processing. To avoid this, the application must set a safe threshold for the visibility timeout.

Sit with how bad this is, because it is the classic production incident with this design. The first worker is still running, doing real work. A second worker picks up the same message and does it again. Nothing has failed and nothing errors — you simply did the job twice.

Worse, it is self-amplifying: the situation arises when processing is slow, which usually means the system is under load, which makes processing slower, which produces more duplicates, which consumes more capacity. A slow consumer becomes a duplicate storm.

Two defences. Size the timeout above the p99.9 of processing time, not the average — the tail is what trips this, exactly as the non-functional characteristics material argued. And where the platform supports it, extend the timeout mid-processing (a heartbeat, SQS's ChangeMessageVisibility) so a long job keeps its claim rather than betting on an upfront guess.

And this is precisely why Lesson 2's idempotency is mandatory rather than advisory. The system will hand the same message to two workers. Idempotency is what makes that survivable.

Comparing the two mechanisms

Same requirement, two very different systems:

The deciding question is usually "is this a stream or a work queue?" A stream has multiple independent readers and value in history — offsets. A work queue has jobs that need doing once each, with independent retry — visibility timeout.

Note the fifth row especially: with offsets, a single poison message sits at an offset everyone must pass, so a failure there blocks the partition or forces you to skip it. Visibility timeout gives per-message granularity for free, which is why dead-letter queues fit it so naturally.

Dead-letter queues

"How should we handle messages a consumer fails to consume after the maximum number of retry attempts?"

A dead-letter queue handles messages that aren't consumed after the consumer has made the maximum number of processing attempts.

It also holds messages that can't be processed successfully due to:

• The messages are intended for a queue that doesn't exist anymore.
• The queue length limit is exceeded — although this would rarely occur with our current design.
• The message expires due to per-message time-to-live (TTL).

A dead-letter queue is also important for determining the cause of failure and identifying system faults.

The three causes are diagnostically different — treat them differently

They arrive in the same place but mean entirely different things:

• Max retries exceeded → usually a poison message: a payload the consumer cannot parse or a record that violates an invariant. Fix the data or the code.
• Queue no longer exists → a coordination failure: something was deleted while producers still referenced it. Fix the deployment order.
• TTL expired → a capacity failure: consumers could not keep up. Fix by scaling consumers, not by touching the message.

Same destination, three different on-call responses. If your DLQ alert does not distinguish them, the first thing anyone does during an incident is work out which one they are looking at — so attach the reason as metadata when the message is routed there.

Interview signal by level

Next: the non-functional side.