Requirements: What to Build, and What to Refuse
In one line: pushing back on a requirement is part of the answer here, because one of the usual ones cannot be built and another should not be.
The requirements as usually stated
A brief for this system typically lists three functional requirements — replay from the DLQ, shape the rate against downstream health, and let engineers filter or modify messages — and three non-functional ones: exactly-once, multi-tenancy, and progress telemetry.
Two of those need work before you design against them.
Refuse: exactly-once delivery
The requirement is usually written as "exactly-once, or at-least-once with deduplication". The parenthetical is the real answer and the headline is not available.
Exactly-once delivery across an unreliable network is impossible, and the proof is old and simple. The Two Generals problem shows that two parties communicating over a channel that can lose messages can never both become certain the other received a message, no matter how many acknowledgements they exchange. A sender that does not receive an ack cannot distinguish "the message was lost" from "the ack was lost", so it must either retry — risking a duplicate — or not — risking a loss.
So the requirement gets rewritten: at-least-once delivery with idempotent processing. That is not a weaker promise dressed up — it is a different design. It moves the correctness burden onto a deduplication boundary you own, which is a thing you can build and test, rather than a delivery guarantee nobody can provide.
Saying this out loud is a strong signal. Accepting "exactly-once" as written and then quietly building deduplication anyway is the weaker answer, because it leaves the interviewer unsure whether you know.
Narrow: message modification
"Let engineers mutate messages before replaying them" sounds like operator convenience and is a footgun at the payload level.
A mutated message is not the message that failed. Three consequences follow:
It can bypass the validation that rejected it. If a message was dead-lettered because a field was invalid, editing that field and replaying makes the system accept something the original producer never sent.
It destroys the audit trail. Six months later, a record exists that no producer emitted, with no way to reconstruct what actually happened.
It hides the bug. The reason those messages failed is now papered over per-incident instead of fixed at the source.
| Mutation | Allow? | Why |
|---|---|---|
| Routing headers — target queue, partition key | Operational, not semantic; the payload is unchanged | |
| Trace and correlation ids | Needed to distinguish a replay from the original | |
| Replay metadata — job id, attempt, original timestamp | Required | This is what makes a replay auditable at the consumer |
| Payload fields | No, by default | Produces a message no producer sent, and bypasses validation |
| Payload, with explicit provenance | Escape hatch only | Recorded as a new message linked to the original, never as the original |
The distinction to hold: headers are operational, payloads are semantic. Filtering — deciding which messages to replay at all — is unambiguously good and should be first-class. Editing what they say is not.
Accept, with sharpening
Replay from the DLQ to a target. Core. Worth noting the target is not necessarily the original queue — replaying into the main queue mixes replayed traffic with live traffic, which the isolation lesson argues against.
Traffic shaping against downstream health. Core, and the differentiator. This is where most of the design effort belongs.
Multi-tenancy. Core, and it drives the scheduler and the fairness model.
Progress telemetry. Accept, with one honesty: under an adaptive rate the completion estimate is a moving target, because the rate is discovered rather than known. Present it as a projection from the current rate with a visible confidence, not a countdown that implies certainty.
Add: four requirements that decide whether it is operable
These are usually missing from the brief, and each one is the difference between a demo and something you would run.
Pause, resume, and abort. A replay runs for hours. An operator will need to stop it — because the target started degrading, because the filter was wrong, because a higher-priority incident started. Without this the only abort is killing the coordinator, which is how you lose the cursor.
Live traffic priority. Replay must never take capacity from requests a user is waiting on. Without stating it, the natural implementation puts replayed messages in the same queue as live ones and quietly makes the outage worse.
Poison detection during replay. A message that failed originally may fail again. Without a cap, the replay loops on it forever and stalls behind a message that will never succeed. The DLQ solved this for the main queue; the replay path needs its own version.
Message staleness policy. These messages are old. A payment retry from three days ago may still be valid; a price update from three days ago is actively wrong to apply. Someone has to decide per stream, and the honest default is to refuse to replay past a configured age rather than guess.
Explicitly out of scope
Saying what you are not building is as useful as saying what you are.
Not a message broker. This coordinates over existing queues rather than replacing them.
Not a retry mechanism. In-line retries with backoff belong in the consumer, before dead-lettering. A message reaching the DLQ has already exhausted them.
Not a data repair tool. Fixing bad data is a different job with different guarantees. The replay path moves messages; it does not correct them.
Key takeaway
Three of the usual requirements are right, one is not available, and one should be narrowed. Exactly-once delivery is impossible over a lossy channel because a missing acknowledgement is indistinguishable from a lost message, so the requirement becomes at-least-once with idempotent processing and the guarantee moves to a deduplication boundary you own. Mutation belongs at the header level; editing payloads creates messages no producer sent and bypasses the validation that rejected them. And four unstated requirements — pause and abort, live-traffic priority, poison detection during replay, and a staleness policy — are what separate an operable system from a demo.
Next: sizing it.