Traffic Shaping: The Control Loop
In one line: a limit on how many requests are in flight controls a system better than a limit on how many you send per second, because in-flight count reacts to the target slowing down and a rate does not.
Rate limit or concurrency limit
The distinction matters more than it sounds.
A rate limit caps submissions per second. If the target slows from 10ms to 500ms, you keep submitting at the same rate and the number of outstanding requests climbs without bound. The rate limiter is satisfied while the target drowns.
A concurrency limit caps requests in flight. If the target slows, completions slow, in-flight count stays at the cap, and your effective rate falls automatically. The limit self-corrects because it is coupled to the target's actual behaviour.
Little's law is the reason: concurrency = rate × latency. Fix concurrency and rate becomes inversely proportional to latency — which is exactly the behaviour you want and you get it for free.
Use a concurrency limit as the primary control, and a token bucket on top only where the target insists on a contractual requests-per-second ceiling.
The loop
Three parameters, and the third is the one people get wrong.
Floor. Never drop to zero. A limit of zero cannot recover, because you stop generating the traffic that would tell you the target improved. Keep a small trickle — enough to probe.
Ceiling. Never exceed a fraction of the target's known steady-state capacity, even if everything looks healthy. This bounds the damage when the signal is wrong or stale.
Interval. Must be longer than the target's reaction time. If the loop adjusts every second but latency takes ten seconds to reflect a change, the controller is reacting to its own previous decisions and will oscillate. This is the classic control-theory failure and it looks like a system flapping between full speed and stopped for no visible reason.
Probing back up
After backing off, something has to test whether conditions improved. Without it the replay stays at the floor forever after one blip.
Additive increase is the probe: one extra concurrent request per interval. It is deliberately slow, and the slowness is the safety property — you approach the limit gradually and discover it by getting close, not by overshooting.
Two refinements worth mentioning:
Probe more cautiously after repeated backoffs. If the limit has been halved three times in five minutes, the target is genuinely struggling and hammering it back up is wrong. Widening the interval after repeated cuts is a small change with a large effect.
Reset the reference periodically. A gradient-based controller comparing against the best latency it has ever seen will treat a legitimately-changed baseline — after a deploy, or a dependency migration — as permanent degradation. Re-establish the minimum on a slow schedule.
Keeping replay behind live traffic
The requirement added earlier, and the mechanism that implements it.
Shaping alone is not isolation. If replayed messages go into the same queue as live traffic, they compete for the same consumers, and a shaper that is slightly too generous steals capacity from users.
| Approach | Isolation | Cost |
|---|---|---|
| Same queue as live traffic | None — direct competition | Simplest, and it is the outage shape from lesson one |
| Separate replay queue, shared consumers | Partial — consumers can prioritise live | Consumers must implement priority |
| Separate queue and dedicated consumers | Strong | Extra capacity to run; may idle |
| Separate queue, consumers scale down under live load | Strong and elastic | More moving parts |
The second row is usually the right default: a separate replay lane the same consumers read, but only after the live queue is empty. It costs no extra capacity and it makes the priority explicit in the consumer rather than implicit in the shaper's tuning.
The point to make either way: the shaper decides how much, and the lane decides who yields. Those are different mechanisms solving different halves, and a design with only the shaper will eventually starve live traffic when the shaper guesses high.
Circuit breaking on top
The shaper handles degradation. It should not be the only protection against a target that has actually gone down.
If the failure rate crosses a hard threshold, or the target starts refusing connections, the replay should pause itself rather than continue at the floor. Then retry the target periodically with a single probe, and resume when it succeeds.
That is a circuit breaker, and it belongs here because the shaper's job is to find a rate — it is the wrong tool for deciding whether to send anything at all. Conflating them produces a shaper that keeps trickling requests at a corpse.
Key takeaway
Prefer a concurrency limit to a rate limit: fixing in-flight count makes effective rate fall automatically when the target slows, which is backpressure for free via Little's law. Run the loop slower than the signal it reads, or the controller reacts to its own past decisions and oscillates — which looks like an unstable target and is not. Keep a floor so probing can recover, a ceiling so a wrong signal cannot do unbounded damage, and probe upward additively. Then separate the mechanisms: the shaper decides how much, a separate replay lane decides who yields to live traffic, and a circuit breaker decides whether to send at all.
Next: the duplicates the lease model guarantees, and where to absorb them.