Evaluation: Functional Requirements

Why this matters: message deletion is the most interesting operation in the whole API, because the two available mechanisms produce entirely different systems from the same requirement.

Key takeaway

Five operations, all satisfied. The one worth real attention is delete, which has two implementations — offset tracking and visibility timeout — and the choice shapes everything downstream.

Queue creation and deletion

Operation	How it works
Creation	The frontend validates the client's details. The cluster manager assigns servers to the new queue and updates the metadata stores and caches via the metadata service
Deletion	The cluster manager deallocates the queue's storage space and removes the associated data from metadata stores and caches

Send and receive

Producers deliver messages to specific queues. At the backend, incoming messages are sorted by timestamps to preserve order and placed in the queue. Consumers then retrieve messages from these queues.

When the frontend receives a message, it identifies the primary host or cluster where the queue resides — depending on the replication model — and forwards the request to that entity for enqueueing.

Message deletion: two approaches

1. No immediate deletion (offset tracking)

Messages are not deleted upon consumption. Instead, the system maintains message order, and consumers track their own progress (offsets). A background job deletes messages once expiration conditions are met. Apache Kafka uses this model to allow multiple processes to consume the same message stream.

Queue (append-only log)

[ m1 ][ m2 ][ m3 ][ m4 ][ m5 ][ m6 ][ m7 ]
              ^                 ^
        consumer B        consumer A
         offset 3          offset 6

Both read the same messages. Neither removes anything.
A background job deletes older messages once they expire.

Offset tracking is what makes replay possible

The consequence worth naming: because messages are not destroyed on read, a consumer can rewind its offset and reprocess history.

That single property is why Kafka is used for things a queue traditionally is not — rebuilding a downstream database after a bug, backfilling a new service with past events, running a new consumer group over data that already flowed. The queue becomes a durable log you can read repeatedly rather than a channel that empties as you drain it.

It also makes fan-out trivial. Adding a sixth consumer group costs nothing, because nobody's reads interfere with anyone else's — the pub/sub pattern from Lesson 1, implemented by simply not deleting anything.

The cost is storage: you keep messages for the full retention period whether or not everyone has consumed them, and retention becomes a capacity decision rather than a consumption one.

2. Visibility timeout

Messages are not deleted immediately but are made invisible to other consumers for a specific duration (visibility_timeout). The consumer must explicitly delete the message via an API call after processing. If the consumer fails to delete it within the timeout, the message becomes visible again.

The consumer is responsible for the final deletion. This ensures high durability: if a consumer crashes before processing is complete, the message reappears in the queue for another worker to handle.

This mechanism provides at-least-once delivery semantics. If a worker fails, the message eventually becomes visible again, ensuring it is processed at least once.

What if the timeout expires while the consumer is still working?

The message becomes visible, and another worker can receive it — duplicating the processing. To avoid this, the application must set a safe threshold for the visibility timeout.

Sit with how bad this is, because it is the classic production incident with this design. The first worker is still running, doing real work. A second worker picks up the same message and does it again. Nothing has failed and nothing errors — you simply did the job twice.

Worse, it is self-amplifying: the situation arises when processing is slow, which usually means the system is under load, which makes processing slower, which produces more duplicates, which consumes more capacity. A slow consumer becomes a duplicate storm.

Two defences. Size the timeout above the p99.9 of processing time, not the average — the tail is what trips this, exactly as the Non-Functional Characteristics chapter argued. And where the platform supports it, extend the timeout mid-processing (a heartbeat, SQS's ChangeMessageVisibility) so a long job keeps its claim rather than betting on an upfront guess.

And this is precisely why Lesson 2's idempotency is mandatory rather than advisory. The system will hand the same message to two workers. Idempotency is what makes that survivable.

Comparing the two mechanisms

Same requirement, two very different systems:

	Offset tracking	Visibility timeout
Message removed on read	No	No — hidden, then explicitly deleted
Who tracks progress	The consumer (its offset)	The queue (per-message state)
Multiple independent readers	Natural — each has its own offset	Awkward — a message is claimed by one consumer
Replay history	Yes	No — deleted is gone
Per-message retry	Harder — offsets are sequential, so one bad message blocks or gets skipped	Natural — that message alone reappears
Storage cost	Full retention period	Only undelivered messages
Canonical system	Kafka	Amazon SQS

The deciding question is usually "is this a stream or a work queue?" A stream has multiple independent readers and value in history — offsets. A work queue has jobs that need doing once each, with independent retry — visibility timeout.

Note the fifth row especially: with offsets, a single poison message sits at an offset everyone must pass, so a failure there blocks the partition or forces you to skip it. Visibility timeout gives per-message granularity for free, which is why dead-letter queues fit it so naturally.

Dead-letter queues

"How should we handle messages a consumer fails to consume after the maximum number of retry attempts?"

A dead-letter queue handles messages that aren't consumed after the consumer has made the maximum number of processing attempts.

It also holds messages that can't be processed successfully due to:

The messages are intended for a queue that doesn't exist anymore.
The queue length limit is exceeded — although this would rarely occur with our current design.
The message expires due to per-message time-to-live (TTL).

A dead-letter queue is also important for determining the cause of failure and identifying system faults.

The three causes are diagnostically different — treat them differently

They arrive in the same place but mean entirely different things:

Max retries exceeded → usually a poison message: a payload the consumer cannot parse or a record that violates an invariant. Fix the data or the code.
Queue no longer exists → a coordination failure: something was deleted while producers still referenced it. Fix the deployment order.
TTL expired → a capacity failure: consumers could not keep up. Fix by scaling consumers, not by touching the message.

Same destination, three different on-call responses. If your DLQ alert does not distinguish them, the first thing anyone does during an incident is work out which one they are looking at — so attach the reason as metadata when the message is routed there.

Key takeaway

All five operations are satisfied, and delete is where the design decisions live. Offset tracking turns the queue into a replayable log with natural fan-out; visibility timeout turns it into a work queue with per-message retry and at-least-once delivery. The dead-letter queue catches what neither can complete.

Interview signal by level

Level	What a strong answer sounds like
L4	"The consumer deletes the message after it finishes processing."
L5	Explains the mechanism: "receive makes the message invisible for a timeout rather than deleting it — if the consumer crashes it reappears for another worker, which is how we get at-least-once."
Staff+	Chooses between the two and names the trap: "offsets if this is a stream with multiple readers and replay value; visibility timeout if it's a work queue needing per-message retry. And I'd size the timeout off p99.9 processing time and extend it via heartbeat, because a timeout expiring mid-processing duplicates work under exactly the load conditions that caused it — a slow consumer becomes a duplicate storm."

Next: the non-functional side.