The Replay Job as a First-Class Entity
In one line: the estimation put the replay at two to five hours, and nothing that runs for hours can be a request.
The state machine
A replay job is created, runs, is interrupted, resumes, and ends — and each of those transitions has to survive the coordinator being restarted underneath it.
The paused state is the one that must exist and is most often skipped. Two things drive it: an operator deciding to stop, and the system pausing itself because the target has degraded past the point where slowing down is enough. Both need to be resumable from exactly where they stopped.
What the job holds
| Field | Purpose |
|---|---|
| Source DLQ and target | What is being replayed, and to where — not necessarily the original queue |
| Filter predicate | Which subset: failure class, time window, error signature |
| Rate policy | Ceiling, floor, and how aggressively to probe upward |
| Budget | Maximum messages or maximum duration — an automatic abort |
| Cursor / checkpoint | Position per partition; the resumability primitive |
| Counters | Submitted, succeeded, failed, skipped, parked |
| Owner and audit trail | Who started it, when, with what filter |
The budget is worth calling out. A replay with no upper bound is a job nobody can reason about; a replay capped at "one million messages or four hours, whichever first" is one an operator can approve. It also provides a safe automatic abort if something is badly wrong.
Leases, not assignment
Work is distributed by leasing a batch to a worker for a bounded time rather than assigning it.
The lease is what makes worker failure survivable without a coordinator watching liveness. It is also the concrete origin of the duplicates the next lesson deals with: a worker that submits a batch and dies before acknowledging has done the work and left no record, so the batch is handed to someone else and submitted twice.
That is not a flaw to engineer away. It is the at-least-once property arriving exactly where the theory says it must, and the deduplication boundary is the answer.
The cursor must never go backwards
The checkpoint is the resumability primitive and the place correctness is most easily lost.
Advance only on acknowledged completion. A cursor that moves when a batch is leased rather than when it is acked silently skips everything in any batch whose worker died. Losing messages during a recovery operation is the worst available outcome, and it is invisible.
Checkpoint frequently enough to bound rework. Every batch is simplest. The cost of a coordinator restart is then at most one batch per worker re-sent, which the deduplication layer absorbs.
Track a cursor per partition. A DLQ with many partitions is drained in parallel and the partitions progress unevenly, so a single global cursor is either wrong or forces artificial serialisation.
Never let the cursor go backwards. If a stale worker acks a batch from before a resume, that acknowledgement must be rejected rather than rewinding the position. A monotonic guard on the cursor is cheap and prevents an ugly class of bug.
Ordering, and whether to promise it
The question an interviewer will ask, and the honest answer is uncomfortable.
A replayed message is already out of order with respect to live traffic. It failed some time ago; the world moved on; later events for the same entity have very likely been processed since. Replaying it in its original position relative to its DLQ siblings does not restore global ordering, because global ordering is already gone.
| Guarantee | Achievable? | Cost |
|---|---|---|
| Global order across all replayed messages | Would require serialising the entire replay | |
| Order within one partition or key | Serialise per key; drops parallelism within a key | |
| Order relative to live traffic | Live traffic already moved past these events | |
| No ordering guarantee | Requires consumers to be commutative or last-write-wins |
The defensible position: offer per-key ordering, refuse global ordering, and be explicit that ordering relative to live traffic is already lost. Consumers whose correctness depends on ordering need to handle out-of-order arrival anyway — which, for a system that dead-letters at all, they already do.
Key takeaway
A replay runs for hours, so it is a durable state machine with a pause state, a budget that bounds it, and a cursor that advances only on acknowledged completion and never moves backwards. Distribute work by bounded leases rather than assignment, which survives worker death without liveness tracking — and which is the concrete origin of duplicates, since a worker that submits and dies before acking has its batch reissued. Offer per-key ordering, refuse global ordering, and say plainly that ordering relative to live traffic was already lost when the message was dead-lettered.
Next: how fast to push, and what signal to decide it from.