Requirements, Throttling Types, and Placement

In one line: "rate limiting" is usually treated as one behaviour. It is three — hard, soft, and elastic — and picking the wrong one either drops good traffic or fails to protect you.

Requirements

Functional:

• Limit the number of requests a client can send within a time window.
• Ensure request limits per window are configurable.
• Notify the client (via error or notification) when a threshold is crossed.

Non-functional:

The availability requirement is unusually sharp here — the limiter is on every request

Most components can be unavailable and degrade something. A rate limiter sits in front of everything, so its availability is a ceiling on the whole service's availability.

That produces a genuinely hard question, and interviewers ask it: if the rate limiter fails, do you accept requests or reject them?

• Fail open (accept): the service stays up, but is now unprotected — and the limiter is most likely to fail precisely when traffic is highest, which is exactly when protection matters.
• Fail closed (reject): protection holds, but a limiter outage becomes a total outage for a component whose only job was to protect against outages.

The usual answer is fail open, with the reasoning stated: rate limiting is a defence-in-depth measure, and the service has other protections — load shedding, autoscaling, connection limits. Turning a limiter failure into a full outage inverts the component's purpose.

That said, fail closed for anything security-critical — a login endpoint with brute-force protection should not open its doors because a counter store is unreachable. The right answer is often per-endpoint, not global, and saying so is the senior version.

Low latency is why this design caches so aggressively

The limiter is on the critical path of every single request. If the check costs 20 ms, you have added 20 ms to every request in the system — including all the ones that were well within their limit.

That is why Lesson 5 puts throttle rules in an in-memory cache rather than reading the rule database per request, and why Lesson 7 goes further and moves the counter update off the critical path entirely.

The general principle worth naming: anything on every request must be measured in microseconds, not milliseconds. A component that adds latency to 100% of traffic to protect against 0.1% of it has to be very cheap to be worth it.

Types of throttling

Elastic throttling is the most useful and the most dangerous

It is appealing: why reject a request when there is capacity sitting idle? Spare capacity that goes unused is waste.

The danger is that it trains your callers on capacity you did not promise. A client that consistently gets 800 requests through against a 500 limit will build against 800 — and then be broken the day the cluster is genuinely busy and the limit actually bites. You have converted a clear contract into an unpredictable one.

It also couples the limit to system load, which means behaviour changes under exactly the conditions where you most want predictability. Debugging "why did this work yesterday" gets much harder.

If you use elastic throttling, the mitigation is to make the excess visible — return headers showing the nominal limit and that the request was served on spare capacity — so clients cannot silently come to depend on it.

Where to place it

Note: API endpoints are ideal vantage points for rate limiting because all traffic passes through them.

Placement is a subjective decision based on the organization's technology stack, engineering resources, and goals.

Client-side rate limiting is not a security control at all

"Vulnerable to tampering" understates it. Anything running on the client is entirely under the attacker's control — they can patch it, disable it, or skip your client and call the API directly with curl.

So client-side limiting protects against exactly one thing: accidental over-calling by a well-behaved client. That is genuinely useful — it prevents a UI bug from hammering your API, and it saves the user's bandwidth and battery. But it must be treated as an optimization, never an enforcement point.

The rule: client-side limiting reduces load; server-side limiting provides the guarantee. Do both, and never rely on the first for anything that matters.

Middleware is the usual answer, and the reason is reuse

Server-side works, but it puts limiting logic in every service — which means every team implements it, every team gets it slightly wrong, and rules live in N places.

Middleware — an API gateway, a sidecar, a dedicated service — gives you one implementation and one place to configure rules, and every service behind it is protected whether or not its authors thought about it. It also rejects over-limit traffic before it consumes application resources, which is the point of limiting in the first place.

The cost is an extra hop, and the fact that the middleware becomes a shared dependency whose failure affects everything — which is the fail-open question again, now with a bigger blast radius.

Building blocks used

Interview signal by level

Next: making it work across many nodes and data centers.