Detailed Design

In one line: the client identifier builder is the component candidates forget, and it is where most real rate limiters actually go wrong. Deciding who a request is from is harder than counting.

The architecture

Component	Role
Rule database	Stores rules defined by the service owner, such as request limits per time unit
Rules retriever	A background process that monitors the database for changes and updates the cache to ensure rules are current
Throttle rules cache	Stores rules in memory for low-latency access. The limiter checks incoming request IDs against this cache
Decision-maker	Evaluates requests against cached rules using a rate-limiting algorithm
Client identifier builder	Generates a unique ID for each request (e.g. IP address, login ID) to serve as the key for the decision maker to track usage

The client identifier builder is where rate limiters actually fail

Everything else in this design is counting. Deciding whose count to increment is the hard part, and getting it wrong breaks the limiter in both directions.

By IP address: simple and needs no authentication — but it lumps together everyone behind a corporate NAT or a mobile carrier gateway, so one office's traffic exhausts a limit shared by thousands of innocent users. Meanwhile an attacker with a botnet or a pool of proxies has effectively unlimited identities. Too coarse for legitimate users, too easy for attackers to evade.

By login ID or API key: the right granularity, and forgeable only with real credentials — but it only works for authenticated traffic. Login endpoints, signup, and password reset are precisely the endpoints most needing protection and have no user ID yet.

The practical answer is layered identity: API key where present, falling back to IP; and on unauthenticated endpoints, limit on IP plus the attempted account, so a brute-force against one account is blocked regardless of source IP while a shared NAT does not lock out a whole office.

If an interviewer asks how you'd rate limit login attempts, that two-dimensional answer — per-IP and per-target-account — is the one they are looking for.

Why a background retriever instead of reading rules per request

Two constraints from earlier lessons collide: rules must be changeable without a deploy (Lesson 4), and the check must add microseconds, not milliseconds (Lesson 2).

Reading the rule database per request satisfies the first and destroys the second — a database round trip on every request, for data that changes maybe weekly.

The retriever resolves it by moving the freshness cost off the request path: rules live in memory for lookups, and a background process reconciles them with the database. The price is a staleness window — a rule change takes effect after the next refresh, not instantly.

That is almost always the right trade, and it is worth naming the exception: if you need an emergency block on an abusive client to take effect immediately, polling is too slow. Real systems add a push or invalidation path for that case, keeping polling as the steady-state mechanism.

Handling a rejected request

If a request exceeds the limit, the API returns HTTP 429 Too Many Requests. The system may then:

Drop the request and return an error message (e.g. "Service unavailable").
Queue the request for later processing if the limit is due to a temporary system overload.

Request processing

When a request arrives, the client identifier builder extracts the client key and forwards it to the decision-maker. The decision-maker checks the cache for rules and current usage counts. If the request is within the configured limits, it is passed to the request processor.

The decision-maker makes decisions based on the throttling algorithms: hard, soft, or elastic.

Throttling mode	What happens to an over-limit request
Hard	Rejected, and an error response is sent to the client
Soft or elastic	Either served beyond the defined limit, or placed in the queue and served later, as resources become available

429 versus 503 — the diagram uses both, and they mean different things

The recovered design shows two distinct rejection paths, and the distinction is a real HTTP semantic worth getting right:

429 Too Many Requests — you sent too many. The fault is the client's, the limit is per-client, and the client should slow down. This is the rate limiter's own verdict.
503 Service Unavailable — the system is overloaded. The fault is not the client's; it might be well within its personal limit while the service as a whole cannot cope. This is the global-capacity backstop from Lesson 3 firing.

Both should carry Retry-After, which is the single most useful thing you can do for callers — it converts "try again at some point" into a schedule, and it prevents the retry storm message-queue design warned about.

Returning 503 for a per-client limit tells the client your service is broken when it is actually working correctly and rejecting them specifically. Getting this backwards is common and easy to avoid.

Queueing an over-limit request is only right in one case

The design allows queueing rather than rejecting, and the design is precise about when: if the limit is due to a temporary system overload.

That qualifier is load-bearing. If a client exceeded its quota, queueing is wrong — you are holding memory for work you already decided the client is not entitled to, and the client sees latency instead of a clear signal to back off. Worse, under sustained abuse the queue grows without bound, which is message-queue design's exact warning: a queue buys time, not capacity.

Queue when the work is legitimate and capacity is temporarily short. Reject when the client is over its entitlement. Same over-limit request, opposite handling, decided by which limit fired.

Key takeaway

Rules live in a database, are pulled into an in-memory cache by a background retriever, and the decision-maker evaluates against them using an algorithm. The client identifier builder decides whose count to increment — the hardest part, and where IP-versus-identity trade-offs bite. Rejections return 429 for client-fault and 503 for system-fault, both with Retry-After.

What to send back, beyond the status code

Rejecting a request is the easy half. Telling the client how to behave is what stops a limiter from causing the retry storm it exists to prevent.

429 Too Many Requests plus Retry-After is the minimum. Without Retry-After a client has no basis for choosing a delay, so it either retries immediately — amplifying exactly the overload you were shedding — or backs off arbitrarily and wastes its own allowance.

Send the limit state on successful responses too, not only on rejections. Headers carrying the ceiling, what remains, and when the window resets let a client pace itself before it hits the wall. A client that can see it has 3 of 1000 requests left will slow down; one that only learns at rejection cannot.

That is the difference between a limiter that shapes traffic and one that merely blocks it. Rejecting without guidance converts a load problem into a retry storm, which is why the headers are part of the design rather than a nicety.

Note also that 429 and 503 say different things: 429 means you asked too often, 503 means we are unwell. Returning the wrong one sends the client down the wrong recovery path.

Interview signal by level

Level	What a strong answer sounds like
L4	"Store the limits in a database and check each request against them."
L5	Caches the rules: "a background process pulls rules into an in-memory cache so the check doesn't hit the database per request, and over-limit requests get a 429."
Staff+	Leads with identity: "the hard part is the identifier. IP is too coarse — a corporate NAT shares one limit across thousands — and too easy to evade with a proxy pool. API key is right where we have one, but login endpoints have no identity yet, so there I'd limit on IP and target account together. And I'd return 429 for client-fault versus 503 for system overload, both with Retry-After, since they tell the caller completely different things."

Next: the concurrency bug that makes counters undercount.