Optimizing the Critical Path

In one line: this is the optimization that makes rate limiting cheap enough to run on every request — and it deliberately accepts inaccuracy to get there. Knowing what you gave up is the point.

Online check, offline update

The check, as the design expresses it:

if (Request(ID).Count < Max. Limit) {
    Request(ID).Allowed = True
    Request.Count++
}

Worked example

The table shows requests made by each client and the maximum allowed per unit time:

Assume a request arrives with request(ID) = 101. The limiter evaluates request(ID).Count <= Max Limit against cached data — 3 is less than 5, so the request is allowed and an Allowed signal is sent immediately to the front-end server.

The usage counters are then updated asynchronously (offline). The sequence:

1. The request is allowed, since 3 is less than 5 for ID 101.
2. The count and other relevant data are updated for the client with ID 101.
3. After specific intervals, the data is written back to the cache.

This approach reduces latency and contention on the critical path.

The insight is that the decision doesn't need the write to have happened

Reading the counter is required to decide. Writing it back is not — it only matters for the next request.

So the write can be moved off the path. The request gets its answer after one cache read; the bookkeeping catches up afterwards. You have removed a write from every request while keeping the read that the decision actually depends on.

This is the same restructuring as distributed caching's offline eviction, and for the same reason: an operation that looked like it had to be synchronous turned out to be a read that only appeared to need a write. When something is on a hot path, ask which half of it the answer actually depends on.

Be precise about how much overshoot this allows

"A small number of excess requests" deserves quantifying, because an interviewer will push.

Between a decision and its write-back, every concurrent request reads the same stale count. If the write-back interval is T and the client sends at rate r, up to r × T requests can be allowed on a single stale value. Client 101 at count 3 with a limit of 5 could get many more than 2 further requests through if they all arrive inside one interval.

So the overshoot is bounded by arrival rate times staleness window, not by some fixed small number. That is fine when the limit protects capacity — and unacceptable when it is a billing quota or brute-force defence.

Notice this compounds with Lesson 3's distributed-counter overshoot. If you run N limiter nodes with local counters and offline updates, the errors multiply. Stacking both optimizations without saying so is a real design mistake, and naming the interaction is a strong senior signal.

This reintroduces the race from Lesson 6 in a different form

Lesson 6 fixed lost updates with atomic operations. This optimization deliberately steps back from that: the decision is now made against a value that may be stale by design.

The difference is intent. The Lesson 6 race was a bug — an unintended lost update. This is a chosen trade: bounded inaccuracy in exchange for removing a write from the critical path.

The two are compatible if you keep the update atomic even when asynchronous — the write-back should still use INCR rather than a read-modify-write, so the offline path does not silently lose counts on top of the deliberate staleness. Accuracy you gave up on purpose is a trade; accuracy you lost by accident is a bug.

TCP flow control works the same way

Note: TCP flow control uses a similar mechanism. The receiver throttles the sender by advertising a "window size" (the amount of data it is willing to receive).

The parallel is exact and worth carrying. The receiver does not approve each segment individually — that would mean a round trip per segment. It grants an allowance in advance and lets the sender proceed until the allowance is exhausted, then updates the window.

Same structure here: the limiter grants permission from cached state rather than confirming against authoritative state per request. Both accept bounded overshoot in exchange for removing a synchronous round trip from every operation.

It is a good analogy to deploy in an interview, because it shows the pattern is not a hack — it is how the most widely deployed flow-control mechanism in computing works.

What the design achieves

The distributed rate limiter meets the non-functional requirements:

The latency the limiter adds

Every check is a network round trip to the counter store, and it sits on 100% of requests — including the ones that pass. That makes the limiter's own latency a first-class concern rather than an afterthought.

Connection pooling is the largest single win and the easiest to forget. Opening a TCP connection per check adds a handshake to every request, which can dwarf the counter operation itself. Persistent pooled connections remove that entirely.

Co-location matters more than raw store speed. A sub-millisecond counter operation reached across a region is dominated by the network, so the limiter and its store belong in the same failure domain and the same region — which is also the argument for regional counters rather than one global one.

And the check itself should be one round trip, not several. An algorithm needing read-then-write doubles the latency and reintroduces the race; a single atomic operation or script does both in one hop, which is why the atomic form is a latency decision as much as a correctness one.

Interview signal by level

Next: the algorithms that make the decision.