Distributing the Limiter

In one line: a single rate limiter cannot handle traffic for millions of users, so the counters must live somewhere shared — and every option trades accuracy against latency. This is the hardest part of the design.

Two models for counter state

Sticky sessions in the load balancer can enforce limits per node — by sending consumers to exactly one node — but reduce fault tolerance and scalability.

Quantify the distributed model's inaccuracy — it is usually acceptable

"Clients may momentarily exceed limits" sounds vague. Bound it and the trade becomes obvious.

With N limiter nodes each holding local state and syncing periodically, the worst case is that a client hits every node just before a sync and gets close to N times the limit for one sync interval. With 10 nodes and a 500/min limit, a determined client might briefly achieve something approaching 5,000.

Now ask what the limit is for. If it exists to stop a service being overwhelmed, and the service can handle 50,000/min, then a brief 5,000 from one client is irrelevant — you got 99% of the protection for a fraction of the latency. If the limit is a billing quota or a brute-force defence, that overshoot is unacceptable and you need the centralized store.

The senior framing: choose the model from what the limit protects, not from a general preference for accuracy. Protection limits tolerate approximation; billing and security limits do not.

Sticky sessions look like a free fix and are not

Route each client to exactly one node and its counter is local, exact, and fast — no coordination at all. Tempting.

The costs the design names are both serious. Fault tolerance: when that node dies, the client's counter dies with it, so the client's budget silently resets — and an attacker who can trigger node failures gets a fresh quota each time. Scalability: load is now distributed by client identity rather than by traffic, so a single heavy client cannot be spread across nodes, and rebalancing after a node change moves counters around and loses them.

There is a third: it constrains the load balancer for everyone. Sticky routing is a global property of the traffic layer, adopted to serve one component's convenience.

It is a reasonable choice for a small, homogeneous client population. It is a poor one at the scale that motivated distributing the limiter in the first place.

Global versus per-user counters

Beyond storage architecture, we must decide between global counters shared by all requests or individual counters per user. For example, the token bucket algorithm can use a single bucket for total traffic or assign specific buckets to users. This choice depends on the use case and rate limiting rules.

The honest answer is usually both, at different layers

"How would you choose between a single counter shared by all users and separate counters per user?"

They solve different problems and neither substitutes for the other.

A global limit protects the service from aggregate overload — but provides no fairness, so one client can legitimately consume everyone's budget and starve the rest.

Per-user limits provide fairness — but a million users each within their individual limit can still collectively exceed what the service can handle. Per-user limits alone do not protect capacity.

So production systems layer them: per-user limits for fairness, plus a global limit as the backstop for total capacity. Naming both, and explaining that they protect different things, is a much stronger answer than picking one.

The trade the question is really probing: per-user costs you state proportional to your user count and the machinery to identify clients reliably — which Lesson 5's client identifier builder exists to do.

Can a load balancer do this instead?

Load balancers help prevent application servers from being overwhelmed by excessive requests. They enforce this by rejecting requests that exceed configured thresholds or forwarding them to a backend queue for deferred processing. Load balancers route requests without considering the computational cost or expected processing time of individual operations.

Certain operations within a web service may be rapid, while others may be more time-intensive. When controlling requests for specific operations, a more effective approach is to implement this control at the application server level with a rate limiter. The rate limiter excels at understanding the intricacies of individual operations and can selectively impose limitations as required.

'Not all requests cost the same' is the whole argument — and it generalizes

A load balancer counts requests. It cannot distinguish a cached profile lookup costing 2 ms from a report generation costing 8 seconds. Both are one request.

So a client sending 100 cheap requests and a client sending 100 expensive ones look identical to the load balancer, while consuming radically different resources. A limit tuned for the expensive case throttles the cheap client pointlessly; one tuned for the cheap case fails to protect against the expensive one.

The rate limiter sits where the operation is known, so it can apply different limits per endpoint — or, in more advanced systems, count cost units rather than requests, charging an expensive call more of the budget. That is how API quotas work at GitHub and Stripe, and mentioning it shows you understand what "request" is actually measuring.

Same shape as load balancing's least-connections-versus-round-robin discussion: counting requests is only a good proxy for load when requests are uniform.

The multi-data-center problem

"A client sends requests from two VMs, one connected via VPN to a different region. The limiter identifies clients by user credentials, so both sessions share the same user ID, and requests may be routed to different data centers. How should rate limiting be enforced?"

If rate limiting is applied independently per data center without global coordination, users could potentially exceed the limit by distributing requests across multiple data centers. To prevent this, a globally shared rate limiter (e.g. a distributed token bucket) would be needed to enforce a consistent limit across all data centers.

This is the accuracy-versus-latency trade at its most extreme, and there is no free answer

Within a data center, the coordination cost is sub-millisecond. Across regions it is tens to hundreds of milliseconds, and the paths are neither fast nor redundant. So a globally consistent limit means every request pays a cross-region round trip before reaching the server that would have handled it locally.

That is usually unacceptable. Which leaves three real positions:

1. Per-DC limits, accept the overshoot. A client hitting K data centers gets up to K times the limit. If the limit protects capacity and each DC is independently protected, this is often fine — each DC is defended, which was the point.
2. Divide the global budget. Give each DC limit / K. Globally correct, but wasteful: a client hitting one DC exclusively gets only its fraction, so most clients are under-served to constrain a rare one.
3. Global coordination only where it matters. Local approximate limits on the hot path; a globally consistent store for billing quotas and security-critical endpoints, where correctness beats latency.

Option 3 is what large systems do, and stating it that way — approximate locally, exact where it counts — is the answer that shows you understand why the question is hard rather than reciting both options.

Interview signal by level

Next: the architecture and how rules are expressed.