High-Level Design: Frontend and Metadata Service

In one line: the frontend is where every request is validated, authenticated, and deduplicated, and the metadata service is what turns a queue name into a physical host. Neither stores a single message, and both are essential.

Two assumptions

Sequence numbers as partition keys is a neat reuse

Lesson 4 established that every message needs a sequence identifier for ordering. This assumption points out that the same identifier can serve as the partition key in a key-value store — key-value stores's building block, dropped in whole.

One value, two jobs: it orders messages and it places them. That is a small thing, but it is the kind of composition that makes a design feel considered rather than assembled — and it is why the sequencer chapter comes before this one.

The architecture

Load balancer

The load balancer layer distributes requests from producers and consumers to frontend servers. This minimizes latency and ensures high availability.

Frontend service

Stateless machines distributed across data centers, performing six functions:

Stateless is what makes the load balancer's job easy

Because the frontend holds no per-client state, any frontend server can serve any request. That means the load balancer needs no session affinity, a failed frontend costs nothing but the in-flight requests, and scaling out is purely a matter of adding machines.

Compare it to the backend, which is inherently stateful — it holds the messages. Two lessons from now you will see how much machinery that requires: cluster managers, primary election, replication protocols.

The general pattern, and it is worth naming in an interview: push state to as few tiers as possible, and make everything in front of them stateless. The Load Balancers chapter made the same argument.

Deduplication here is a different guarantee from consumer idempotency — don't conflate them

Frontend deduplication rejects an identical request arriving twice at the frontend — a producer that retried because it never saw the response, for instance. It is checked against a hash key store before the message enters the queue.

That does not make delivery exactly-once. Lesson 2's duplicate arises after the message is safely in the queue, when a consumer processes it and dies before acknowledging. No frontend check can see that, and redelivery is deliberate.

So the two mechanisms guard different ends of the pipeline:

• Frontend dedup → stops the same message being enqueued twice.
• Consumer idempotency → stops an enqueued message being acted on twice.

You need both, and neither substitutes for the other. Claiming exactly-once because you deduplicate at the frontend is a common and detectable error.

Metadata service

Manages queue metadata in the metadata store and cache. It updates both whenever a queue is created or deleted. Acting as middleware between frontend servers and the data layer, it prioritizes cache lookups; on a cache miss it retrieves from the store and updates the cache.

That is read-through caching, from distributed caching

Check the cache, miss, fetch from the store, populate the cache, return. And on write — queue creation or deletion — update both, which is write-through.

Write-through is the right call here for a reason worth stating: metadata is low-volume and high-stakes. Queue creation happens rarely, so the extra write latency is irrelevant, and stale queue metadata would route messages to a host that no longer owns the queue. This is the exact situation that chapter identified as write-through's home ground.

Organizing the metadata cluster

Two approaches, depending on whether the metadata fits on one machine.

1. Small metadata

If it fits on a single machine, replicate it across all cluster servers. Requests can be served by any server, and a load balancer can sit between frontend servers and metadata services.

2. Large metadata

If metadata is too large for a single machine, shard it. Two strategies:

This is the same choice the cache client faced, one tier down

Frontend mapping puts routing knowledge in the caller — one hop, lowest latency, but every frontend must be kept in sync with the shard map. Diverging views send requests to the wrong host.

Host-based mapping lets any host take a request and forward it — two hops, but the mapping lives in one tier rather than being replicated to every client, so keeping it current is far easier.

That is exactly the trade distributed caching made: the cache client hashes locally (fast, but every client must agree) versus routing through an intermediary (slower, but one place to update). Recognizing the repeated shape is the point — client-side routing buys a hop and costs you a consistency problem.

Why host-based "suits read-intensive applications": reads dominate, the extra internal hop is cheap relative to the work, and read traffic tolerates the indirection better than a tight write path would.

Interview signal by level

Next: where the messages actually live.