Progress, ETA, and Operating a Replay
In one line: the replay's rate is the output of a control loop reacting to a service you do not control, so any completion estimate is a projection from current conditions and should be presented as one.
What progress actually means
Four counters, and they must not be collapsed into one percentage.
Matched messages the filter selected the denominator Submitted sent to the target Succeeded acknowledged as processed Skipped excluded as stale, or deduplicated Parked failed repeatedly during replay; poison
Submitted and succeeded are different numbers, and the gap between them is the most useful thing on the page. A replay where submitted is climbing and succeeded is flat means the target is accepting and failing — the exact condition where an operator should intervene, and it is invisible in a single progress bar.
The completion estimate
The naive calculation is remaining divided by current rate. It is wrong in a specific way worth explaining, because the rate is not a constant — it is the output of a controller that will change it.
Remaining 4,100,000 Current rate 600 / sec Naive ETA ~1.9 hours
That number is only true if the rate holds, and the rate is a live negotiation with a service whose load changes. Three things make it move: live traffic rising and squeezing headroom, the target's dependencies degrading, and the shaper probing upward and finding room.
Three honest presentations, in increasing order of usefulness:
A range from observed rate variance. "Between 1.5 and 3 hours at current rates."
A projection with an explicit assumption. "About 2 hours if the current rate holds" — stating the condition rather than implying certainty.
A rate chart alongside the estimate. The most useful. An operator seeing the rate climbing steadily reads the ETA as pessimistic; seeing it collapse repeatedly tells them more than any single number.
What to alert on
Progress is for watching. Alerts are for the conditions where nobody is.
| Condition | Why it matters |
|---|---|
| Success rate falling below a threshold | The target is failing what it accepts |
| Rate at the floor for an extended period | Effectively stalled; the target is not recovering |
| Circuit breaker open beyond a duration | The target is down, not slow |
| Parked count climbing fast | Systematically bad messages, not a bad target |
| Job exceeded its expected duration | Estimate was wrong or something is stuck |
| Live traffic latency degrading during a replay | The replay is harming users — the most important one |
The last row is the one that closes the loop with the shaper. The shaper should already be reacting, and an alert here is a check on the shaper itself. If live latency is degrading while the shaper believes conditions are healthy, the shaper is reading the wrong signal — and that is a failure no other alert catches.
The operator actions
Each one exists because of a specific situation, and a design missing any of them has a gap an interviewer can find.
Pause and resume. A higher-priority incident starts, or the target begins degrading. Must resume from the checkpoint, not restart.
Abort. The filter was wrong, or the replay is no longer worth finishing. Stops cleanly and leaves the remaining messages in the DLQ.
Adjust the rate ceiling. An operator with context the shaper lacks — knowing a maintenance window is starting — can lower the ceiling without stopping.
Drain the parked messages. Poison messages accumulate somewhere and need their own review path. They are the replay's own dead-letter queue, and yes, it must be terminal too.
Re-run a filter as a dry run. Before committing to a second replay, see what it would match.
The replay that should be abandoned
Worth stating because designs assume every replay finishes.
Sometimes the right answer is to stop. The messages have aged past usefulness. The consumer bug is not fixed, so every replayed message will fail again. A newer event has superseded every message in the backlog. The business decided the data is not worth the risk.
An abort that leaves messages in the DLQ, with a record of what was attempted, is a successful outcome of the system. A coordinator that only supports running to completion forces an operator to choose between finishing something pointless and killing the process — and killing the process is how the cursor gets lost.
Key takeaway
Report matched, submitted, succeeded, skipped and parked separately — the gap between submitted and succeeded is the signal a single progress bar destroys. Present completion as a projection with an explicit assumption and a rate chart, never a countdown, because the rate is the output of a controller and a countdown that jumps teaches operators to ignore the page. Alert on live traffic degrading during a replay above everything else, since that is the only check on whether the shaper is reading the right signal. And support abandoning a replay cleanly, because a replay that is no longer worth finishing is a normal outcome.
Next: the walkthrough, where all of it gets designed live.