Multi-Tenancy and Isolation
In one line: the shapers are naturally isolated because each targets a different service, and everything else in the coordinator is shared — which is where the contention lives.
What is isolated for free
Each tenant replays into its own target, so each needs its own control loop with its own limit, its own reference latency and its own circuit breaker. There is no interaction: one service being slow says nothing about another.
That is the pleasant half, and it means the hard part is elsewhere.
What is actually shared
The worker pool is the contended resource. A tenant replaying five million messages will happily consume every worker for hours if nothing stops it, and the other three hundred tenants wait.
Fairness
Three models, with a clear answer.
| Model | Behaviour | Weakness |
|---|---|---|
| FIFO by job creation | First replay gets everything until done | A five-hour job blocks everyone behind it |
| Equal share | Every active job gets the same slice | A tiny replay gets as much as a critical one |
| Weighted fair share | Slice proportional to a priority weight | Someone must set weights, and everyone claims to be tier one |
Weighted fair share with a floor is the defensible design: every active job is guaranteed a minimum slice so nothing starves, and remaining capacity is distributed by weight.
The floor matters more than the weighting. Without it, a high-weight tenant during a mass incident starves everyone else completely, and a starved replay is indistinguishable from a broken coordinator to the team waiting on it.
Two refinements worth naming:
Weights come from service tier, not from the requester. Ask teams to self-declare priority and everything is critical. Derive it from an existing service-criticality registry if one exists.
Cap per-tenant concurrency regardless of weight. A tenant with the highest weight and no cap is FIFO with extra steps.
Isolating the failure modes
Fairness handles contention. Isolation handles a tenant behaving badly.
A tenant whose target is a black hole. Every submission times out, consuming a worker slot for the full timeout. Their shaper backs off, which helps — and the circuit breaker is what actually releases the slots, by pausing the job rather than trickling into a corpse.
A tenant with millions of poison messages. Every message fails immediately. Fast failures burn worker slots at high rate and generate error signal that means nothing about the target. Poison detection — parking a message after repeated replay failures — is what bounds this, and it is per-job so it cannot spill.
A tenant with an enormous backlog. Not misbehaviour, just size. The per-tenant concurrency cap handles it, and the budget from the job model gives an automatic stop.
A tenant hammering the dedup store. A hot key range from one job can degrade lookups for everyone. Partition the dedup store by tenant so the blast radius is bounded.
Each mechanism appeared earlier for a single-tenant reason. Multi-tenancy is what makes them non-negotiable, because without them one tenant's problem becomes everyone's.
Where isolation should be structural
A recurring principle from elsewhere in system design applies here too: prefer isolation that cannot be forgotten.
Partition the dedup store by tenant. A tenant prefix on the key gives bounded blast radius essentially for free.
Consider separate worker pools for the highest tier. Dedicated capacity for a handful of critical services costs idle capacity and removes them from contention entirely. Worth it only if the tier is genuinely small.
Never share a replay lane between tenants. Each target gets its own lane. Sharing means one slow target backs up another's replay, which is a coupling with no benefit.
Quotas
Rate limits alone are the wrong unit, for the same reason they are in most multi-tenant systems: they bound instantaneous consumption and not total.
| Limit | Bounds |
|---|---|
| Concurrent replay jobs per tenant | Runaway job creation |
| Worker slots per tenant | Instantaneous share of the scarce resource |
| Messages replayed per day | Total consumption over time |
| Dedup keys stored per tenant | Storage blast radius |
The daily message quota is the one that catches abuse the others miss — a tenant repeatedly replaying the same backlog because their consumer is still broken, consuming capacity indefinitely while making no progress. That pattern is common and looks legitimate at every instant.
Key takeaway
The shapers are isolated for free because each targets a different service; the worker pool is the genuinely scarce shared resource and the thing to design around. Assume the mass incident — a shared dependency failing dead-letters across fifty services and starts fifty replays at once — rather than a few concurrent jobs. Use weighted fair share with a floor, because starvation is indistinguishable from a broken coordinator to the team waiting. And make isolation structural where you can: partition the dedup store by tenant, never share a replay lane, and quota total daily volume as well as instantaneous share.
Next: showing progress on something whose rate is being discovered.