Classification at Failure Time
In one line: a DLQ holding two completely different kinds of failure mixed together cannot be replayed sensibly, and the fix happens at the consumer, hours earlier.
Two failures that look the same and are not
Transient. The downstream service was down, a connection timed out, a dependency was throttling. The message is fine. Retrying later works.
Non-transient. The payload failed to deserialise, a required field is missing, the schema does not match, a referenced entity does not exist. The message is broken. Retrying it a thousand times produces a thousand identical failures.
If both land in one DLQ undifferentiated, every replay is contaminated: the broken messages fail again, consume replay capacity, generate error signals that make the traffic shaper back off, and pollute the very telemetry an operator is watching to decide whether the replay is going well.
Uber's Kafka reprocessing design does exactly this: a failure classified as non-transient — a deserialisation failure, a schema mismatch — skips the retry tiers entirely and is published straight to the dead-letter topic. There is no value in delaying a message that will never succeed.
The tiered retry topology
The structure this sits inside is worth knowing because it is the reference implementation.
Rather than retrying in-line and blocking the consumer, failed messages are published to a separate retry topic with a delay. A message that fails there moves to a topic with a longer delay, and so on. Only after exhausting the tiers does it reach the DLQ.
| In-line retry | Tiered retry topics | |
|---|---|---|
| Blocks the partition | Yes — the batch waits | No — the message leaves the main topic |
| Delay between attempts | Occupies a consumer thread | Costs nothing; the tier's delay does the waiting |
| Live traffic impact | Direct — throughput drops | None |
| Observability | A retry is invisible | Tier depth shows how bad things are |
| Cost | Free | More topics to operate |
The fourth row is underrated. Because each tier is a real topic with a real depth, the shape of the failure is visible before anyone looks at a dashboard — a spike in tier one is a blip, and messages reaching tier three means something is genuinely broken.
Flow control between tiers uses a leaky bucket, which is the same idea the traffic shaper will use later at larger scale.
What the DLQ must record
A DLQ entry that is only the original message is nearly useless for replay. It needs the context of its own failure.
| Field | Why the replay needs it |
|---|---|
| Original message and headers | The thing to resubmit |
| Failure classification | Whether replaying is worth attempting at all |
| Error and exception detail | Grouping millions of failures into a few causes |
| Attempt count and tier reached | Distinguishes a hard failure from an unlucky one |
| First-failure and dead-letter timestamps | Staleness policy, and how old the backlog is |
| Source topic, partition, offset | Provenance, and ordering if it matters |
| Consumer version or build | Whether a deploy caused this |
The consumer version is the one people omit and the one that answers the most common question during an incident: did our release cause this? If every message in the DLQ carries the build that rejected it, the answer is a group-by rather than an investigation.
The DLQ must be terminal
A specific and damaging misconfiguration, worth knowing because it is easy to make and silent.
Message brokers let you attach a redrive policy to any queue — including a dead-letter queue. If you do, a message that fails while being consumed from the DLQ gets dead-lettered out of it. If nothing is configured beyond that, it is gone.
The rule: a dead-letter queue is the last stop. Nothing leaves it except through a deliberate, recorded replay. This matters more once a coordinator is reading from it under load, because that is exactly when transient failures during reads become common.
What this buys the replay
Classification at failure time changes what the replay can do:
Skip what cannot succeed. Non-transient failures are excluded by default rather than discovered by failing again.
Group and filter meaningfully. "Replay everything that failed with connection-refused" is a filter an operator can reason about. "Replay messages 4,000,000 through 5,400,000" is not.
Interpret the shaper's signals correctly. If the only failures during replay are transient, a rising error rate genuinely means the target is struggling — rather than meaning you are re-sending broken messages. Without classification, the shaper cannot tell those apart and will throttle for the wrong reason.
That last one is the connection to the next two lessons, and it is why this one comes first.
Key takeaway
Transient and non-transient failures are indistinguishable once they are mixed in one queue, so classify at the consumer, at the moment of failure: broken messages skip the retry tiers and go straight to the DLQ, because delaying something that will never succeed buys nothing. Tiered retry topics keep retries off the main partition and make failure depth visible. Record why a message died, not just that it did — including the consumer build. And never attach a redrive policy to a DLQ; it is terminal, or messages are lost out the back of it.
Next: modelling the replay itself as something that can be paused, resumed and audited.