Traffic Shaping: Choosing the Signal
In one line: you cannot ask the target how much replay it can absorb, because it does not know — so the rate has to be discovered from a signal, and the choice of signal decides whether you find out before or after you break it.
Why a configured rate does not work
The obvious design is a number: replay at 500 messages per second.
It fails in both directions. Too high and it re-triggers the outage the replay was meant to resolve. Too low and a five-hour drain becomes fifteen, and someone raises it manually at 3am under exactly the conditions where judgement is worst.
And there is no correct constant, because the target's spare capacity is not constant. It varies with live traffic, with how warm its caches are, with what its own dependencies are doing. Capacity is a moving quantity, so the rate has to be a control loop rather than a setting.
The signals, and when each one tells you
That ordering is the lesson. Errors are a trailing indicator: by the time a service returns them it is already failing, and some user request has already been harmed. Latency is a leading indicator: queues form inside a service before anything is rejected, so response time degrades while everything is still nominally succeeding.
A shaper that reacts only to errors is always late, and its correction always arrives after the damage.
| Signal | Leads or trails | Weakness |
|---|---|---|
| Error rate | Trails badly | Damage already done; and replay errors may be the messages, not the target |
| Latency (p50, p99) | Leads | Needs a baseline; noisy at low volume |
| Queue depth at the target | Leads | Requires the target to expose it |
| Consumer lag on the main topic | Leads | Excellent when available — directly measures falling behind |
| Explicit backpressure signal | Leads, and unambiguous | Requires the target to cooperate; rare across teams |
The fourth row deserves emphasis in a messaging system. Consumer lag is the ideal signal where it exists: if the target's own consumers start falling behind on the main topic, replay is taking capacity from live traffic, and that is precisely the condition to back off on. It measures the thing you actually care about rather than a proxy for it.
Borrowing from congestion control
The problem — push as fast as possible through a path whose capacity is unknown and varying, without collapsing it — is TCP congestion control. That work is decades old and the algorithms transfer directly.
AIMD
Additive increase, multiplicative decrease. Increase the limit by one per round trip while things are healthy; halve it the moment they are not.
The asymmetry is deliberate and it is the important part. Recovery from overshooting is fast because the cut is large; approach is slow because the increase is small. A system that increased and decreased symmetrically would oscillate around the limit, spending half its time over it.
healthy -> limit = limit + 1 degraded -> limit = limit / 2
Simple enough to reason about at 3am, which matters more than optimality for something that runs during incidents.
The gradient algorithm
The more precise approach, from TCP Vegas: infer capacity from the ratio of the best latency you have seen to the current latency.
gradient = min_RTT / current_RTT gradient near 1 -> no queueing, headroom exists, push more gradient falling -> queues are forming, back off proportionally
This is a genuinely better signal because it detects queueing before any error and scales the response to the severity. Its weakness is that min_RTT needs re-establishing — a service whose baseline legitimately shifts after a deploy will look permanently degraded until the minimum is re-measured.
What to compare against
A shaper needs a reference for "healthy", and there are three options with different failure modes.
A configured SLO. Simple, and stale — nobody updates it, and it does not know about today.
A learned baseline from the target's own history. Adapts, and it will happily learn a degraded state as normal if the replay starts during a bad period.
Live traffic, measured concurrently. The strongest. Compare the latency the target is giving live requests now against what it gave them before the replay started. If live latency degrades, the replay is causing harm — regardless of what the replay's own responses look like.
The third one is worth arguing for explicitly, because it measures the thing that actually matters. The replay's own latency could be fine while live traffic suffers, if the two use different paths or priorities. Watching live traffic answers the real question: is this replay hurting users?
Key takeaway
Capacity is not constant, so the rate must be a control loop rather than a configured number. Choose a leading signal: latency rises as queues form inside a service, while errors only appear once it is already failing — so an error-driven shaper is always late. Consumer lag is the ideal signal where it exists, because it directly measures whether replay is taking capacity from live traffic. Borrow AIMD from congestion control for its deliberate asymmetry — slow approach, fast retreat — and compare against live traffic's latency rather than a configured SLO, because that answers whether the replay is hurting users.
Next: turning the signal into an actual rate.