The DLQ Is Not the Hard Part

In one line: every message broker can already move a failing message aside, and none of them will tell you how to put a few million of them back.

What is already solved

A dead-letter queue is standard and shallow. A consumer fails a message some number of times, the broker moves it to a separate queue, and the main queue stops being blocked by it. In one common implementation this is a single maxReceiveCount setting.

That solves exactly one problem — a poison message blocking a partition — and it is worth being precise that this is all it solves. The DLQ is a holding area. Nothing about it says what happens next.

Why putting them back is dangerous

The naive move is to redrive the DLQ into the main queue and let the consumers work through it. At small volumes this is fine. At scale it is the mechanism that causes a second outage.

The service has just recovered. Its caches are cold, its connection pools are empty, its JIT is unwarmed, and any dependency it calls is in the same state. Presenting it with a backlog of a few million messages at maximum consumer throughput is presenting it with a load spike far above steady-state traffic, at the moment it is least able to absorb one.

Google's SRE material puts a number on the consequence: cascading failures take three to five times longer to recover from than isolated ones, and bringing services back online without care pushes them straight back into overload. The replay is not a data-movement task. It is the thing most likely to cause the second incident.

The shape of the problem

Notice what kind of system this is. There is very little novel storage and almost no interesting data modelling. What there is:

A control loop. Something must decide, continuously, how fast to push — from signals about a target whose capacity is unknown and changing.

A long-running, resumable job. A replay of millions of messages runs for hours. It must survive coordinator restarts, be pausable, and never lose or double-count its position.

A correctness boundary. A replayed message may already have been partially processed. Replaying it must not charge a customer twice.

A multi-tenant scheduler. One coordinator serves hundreds of services, each with its own DLQ, its own target, and its own idea that its replay is the urgent one.

An answer that spends its time on queue technology has misread the question. The interesting parts are the four above, and the first is the one that distinguishes this problem from every other messaging design.

Where the design pressure comes from

Three properties make this harder than it looks, and they are worth stating early because the rest of the chapter keeps returning to them.

The target's capacity is unknown and moving. You cannot ask a service how much replay it can absorb. It does not know. Capacity changes with its own live traffic, its dependencies, and how warm it is — so the rate has to be discovered continuously rather than configured.

Live traffic and replay traffic compete for the same service. Replay is by definition lower priority than the requests users are waiting on, and both arrive at the same consumers unless you deliberately separate them.

The messages are old. A message in a DLQ failed at some point in the past. By the time it is replayed, the world may have moved on — the order was cancelled, the price changed, a later event already superseded it. Replaying blindly is not obviously correct.

Next: which requirements to accept, and which to refuse.