High-Level Design and Rule Format

In one line: the rule format is worth more attention than it usually gets. Rules as data rather than code is what makes a rate limiter operable — and it is a small, concrete example of a principle that shows up everywhere.

The request flow

Step by step:

A request with ID 101 is received by one of the web servers.
The request is forwarded to the rate limiter.
If the request is allowed, the corresponding count is incremented.
The web server forwards the request to one of the API servers.
The response is sent back to the web server after serving the request.

Note that the limiter is consulted, not traversed

The request does not flow through the rate limiter — the web server asks it for a decision and then forwards the request itself.

That is a meaningful architectural choice. The limiter never touches the request body, never proxies the response, and can be scaled independently of request size or duration. It answers a small question with a small answer.

It also means a limiter failure is a failed lookup, not a broken connection — which is what makes the fail-open option from Lesson 2 practical to implement. If the limiter were a proxy in the path, "fail open" would mean bypassing a component that holds the socket, which is much harder.

Same shape as distributed caching's client library: keep the decision-maker off the data path and it stays cheap.

Rules as configuration

Lyft's rate-limiting service provides a practical example of how these rules are structured:

yaml

domain: messaging
descriptors:
  - key: message_type
    value: marketing
    rate_limit:
      unit: day
      requests_per_unit: 5

In this example, the unit is set to day and requests_per_unit is 5. This rule limits the client to five marketing messages per day.

Rules as data is the design decision hiding in this YAML

The rule is configuration, not code. That has three consequences worth naming:

Changeable without a deploy. Lesson 2's functional requirement said limits must be configurable, and Lesson 1 explained why: pricing tiers and quotas change on a business timeline. A rule in a config file or database can change in seconds; a constant in code needs a release.

Composable by descriptor. The key/value pair means rules match on attributes of the request — message type here, but equally endpoint, tenant tier, or API key. You can express "marketing messages: 5/day, transactional: unlimited" without writing branching logic.

Auditable. Rules in a store can be listed, diffed, and reviewed. Limits scattered through application code cannot — and nobody can answer "what are our current limits?" without grepping.

This is why Lesson 5's architecture has a rule database and a rules retriever rather than a hardcoded threshold. The YAML is small; the architectural implication is not.

The descriptor model generalizes to hierarchical limits

domain: messaging plus a descriptors list is a scoping mechanism, and real deployments stack them: a limit on the domain overall, tighter limits per descriptor within it.

That maps directly onto Lesson 3's global-versus-per-user discussion. A domain-level rule is the global backstop protecting capacity; descriptor-level rules are the per-client fairness limits. One config format expresses both, and a request is checked against every matching rule, failing if any is exceeded.

If an interviewer asks how you'd support "1000/min per tenant but also 50/min per tenant on the expensive export endpoint," this structure is the answer — two descriptors, both evaluated.

What the high-level design leaves open

The high-level design leaves key questions unanswered:

Where are the rules stored?
How do we handle rate-limited requests?

The second question has more than one right answer, and that's the point

"Reject it" is the obvious response and it is incomplete. Lesson 5 shows the design supports rejecting or queueing — and which one is correct depends on why the request was limited.

If the client exceeded its quota, rejection is right: the client should back off, and queueing would just hide the problem while consuming memory.

If the limit fired because the system is temporarily overloaded, queueing may be right: the work is legitimate and capacity will return shortly.

That distinction — client fault versus system fault — is what the throttling types in Lesson 2 encode, and it is why hard, soft, and elastic throttling produce different handling for the same over-limit request.

Key takeaway

The limiter is a dedicated service the web server consults rather than a proxy the request traverses, which keeps it cheap and makes fail-open practical. Rules live as configuration matched on request descriptors, which is what makes limits changeable without a deploy and composable into hierarchical policies.

Getting a rule change into a running fleet

Rules live in a database, but the limiter cannot read that database on every request — that would put a second network hop on the hot path and make the rule store a availability dependency of every API call.

Rules are cached in memory and refreshed out of band. A background worker pulls changes on an interval, or subscribes to a change feed, and swaps the in-memory copy. The request path only ever reads local memory.

Two consequences worth stating. There is a staleness window equal to the refresh interval, so a limit tightened in response to an incident takes effect on the order of seconds rather than instantly — and if that matters, you need a push channel rather than polling.

And the limiter must keep working when the rule store is unavailable, serving the last known rules. A limiter that fails because it cannot re-read configuration has turned a config dependency into an outage, which is precisely the fragility it was deployed to prevent.

Interview signal by level

Level	What a strong answer sounds like
L4	"The request hits the rate limiter first, which allows or blocks it."
L5	Separates decision from data path: "the web server asks the limiter for a decision rather than proxying through it, so the limiter stays cheap and independent of request size."
Staff+	Argues for rules as data: "limits change on a business timeline — pricing tiers, quotas — so they belong in a store matched on request descriptors, not as constants in code. That also lets us stack a domain-level backstop with per-tenant and per-endpoint rules in one format, and it's what makes the limits auditable."

Next: the components that make it work.