Estimation

In one line: two calculations decide this design — how big the backlog gets, and how long it takes to drain at a rate the target survives.

How much lands in the DLQ

Start from the brief's scale and a plausible outage.

Total events                 = 5 billion / day
                             = ~58,000 / sec average
                             = ~150,000 / sec at peak (3x)

One service handles          ~2% of that       = ~3,000 / sec
It is down for               30 minutes        = 1,800 s
                                               ----------
Messages dead-lettered       3,000 x 1,800     = ~5.4 million

Five and a half million messages from a half-hour outage of one mid-sized service. That number is the whole problem, and it is worth producing early because it makes everything downstream concrete.

Storage is not the issue:

5.4M messages x ~2 KB      =  ~11 GB

Eleven gigabytes is nothing. The DLQ is cheap to hold and expensive to drain, which is the asymmetry the design has to respect.

How long the replay takes

This is the calculation that matters, and it has a range rather than an answer.

Backlog                       5.4M messages
Target's normal throughput    3,000 / sec

At 100% of normal (reckless)  5.4M / 3,000    =  30 min
At 20% headroom (safe)          600 / sec     =  2.5 hours
At 10% headroom (cautious)      300 / sec     =  5 hours

Two things fall out, and both shape the architecture.

The replay runs for hours, not minutes. So it must survive coordinator restarts, be resumable from a checkpoint, and be pausable. A job model is not gold-plating; it is implied by this arithmetic.

The headroom fraction is the entire design decision. Going from 10% to 20% halves the completion time. Going to 100% completes in half an hour and re-triggers the outage. The traffic shaper exists to find the largest fraction the target actually tolerates, continuously, rather than having someone guess it.

Sizing the coordinator

The pleasant surprise, and worth stating because it redirects effort.

Concurrent replay jobs        ~50 (hundreds of tenants, few active at once)
Replay rate per job           ~600 / sec
Aggregate submit rate         50 x 600      =  30,000 / sec

Thirty thousand submits per second across the fleet is a moderate service. The coordinator does no heavy computation — it leases batches, checks a deduplication key, submits, and records progress.

Dedup store, per replayed message:
  key ~64 B + metadata ~64 B  =  ~128 B
  5.4M messages               =  ~700 MB for one replay
  with a 7-day TTL and several concurrent replays: tens of GB

Tens of gigabytes in a key-value store with TTL. Unremarkable.

So the coordinator is a small system in front of an enormous one, and almost none of the difficulty is in its own capacity. Saying that explicitly is useful, because it tells the interviewer you know where to spend the remaining time — on the control loop and the correctness boundary, not on scaling the coordinator.

The number to check before designing anything

One more, because it determines whether the whole approach is viable:

DLQ growth rate during the outage   3,000 / sec
Safe replay rate after recovery       600 / sec
                                    -----------
Ratio                                     5 : 1

Draining takes roughly five times as long as filling. A one-hour outage produces a five-hour replay.

That ratio is the argument for everything the chapter adds later: it is why replays need scheduling and prioritisation, why a stale-message policy matters — some of those messages will be many hours old before they are attempted — and why an operator needs to be able to abort a replay that has become pointless.

Next: how messages get into the DLQ in the first place, and why that decision matters more than the replay.