Detailed Design: Configuration Service and Replication

In one line: this lesson closes all three gaps from the high-level design, and the configuration-service decision is a clean worked example of choosing the most complex option because it is the only correct one.

Keeping the server list current

Three strategies, in increasing order of both cost and correctness:

Although the configuration service is the most complex and expensive to operate, it is the most robust solution.

Why complexity wins here specifically

The usual advice is to prefer the simpler option. This is a case where the simple options do not actually solve the problem.

Solutions 1 and 2 both require a human to notice a server failed and push an update. During the window between failure and update, clients route to a dead server — and Lesson 5 showed that divergent client views cause silent misses rather than errors.

The configuration service removes the human from the loop, which matters because the failures it handles happen on machine timescales, not human ones. This is the same argument that building block made about four nines requiring automated recovery: once your budget is minutes, a person in the path has already spent it.

Improving availability

To reduce the risk of data unavailability during server failures, the system introduces replica nodes. A typical shard is one primary node and two replica nodes.

To ensure consistency, writes are performed synchronously across replicas when nodes are located in close proximity.

The second benefit is the answer to hotkeys

Replicas are usually justified by availability. Here the read scalability benefit does real work: a hot shard's traffic can be served by three nodes instead of one.

That does not fully solve a single scorching key — all three replicas still hold it, so you have tripled capacity rather than distributed it. But tripling is often enough, and Lesson 10 covers what to do when it is not.

Note also the qualifier on synchronous writes: "when nodes are located in close proximity." Synchronous replication across a data center is affordable; across regions it is not, which is exactly the trade Lesson 10 revisits.

Cache server internals

Three mechanisms, working together:

Cache shard (primary, mirrored on secondary)

Map                          LRU doubly linked list
Key    | Pointer
Key 1  | ffffx1  --------->  HEAD [Value 1] <-> [Value 2] <-> [Value 3] <-> [Value 4] TAIL
Key 2  | ffffx5  ----------------------^
Key 3  | ffffxd  -------------------------------------^
Key 4  | ffffxa  ---------------------------------------------------------^

Primary and replica each run the same internal mechanisms — a replica is a full copy, not a lesser one.

Why there is a delete API

We generally do not expose a delete API because eviction (via algorithm) and expiration (via TTL) are handled locally. However, an explicit delete API is required to remove cache entries when the corresponding data is deleted from the database — to ensure consistency.

The one case TTL cannot handle

Eviction and expiry between them cover almost everything: full cache, stale entry. Neither covers the underlying record being deleted.

If a user deletes their account and the cache holds their profile with a 10-minute TTL, that profile remains servable for 10 minutes after it ceased to exist. For a privacy deletion or a takedown, that is not a stale read — it is serving data you were required to destroy.

Same conclusion as CDNs reached about purge: TTL bounds staleness, and some things cannot wait for a bound.

The detailed design

How it functions:

• Client requests reach the service hosts through the load balancers, where the cache clients reside.
• Each cache client uses consistent hashing to determine which cache server to contact, then forwards the request to the server maintaining that shard.
• Each cache server has primary and replica servers, all using the same internal mechanisms to store and evict entries.
• The configuration service ensures all clients see an up-to-date, consistent view of the cache servers.
• Monitoring services log and report metrics for the caching service.

Cache entries are stored and retrieved from RAM.

Consistent hashing can still distribute unevenly — use virtual nodes

"Consistent hashing may result in unequal distribution of data, and certain servers may get overloaded. How do we resolve this?"

Use a flavor that distributes load uniformly and can make multiple copies of the same data on different cache servers. Each cache server hosts virtual servers internally, with the number depending on the machine's capability. That gives finer control over the load on each server and improves availability.

This is exactly the virtual-nodes mechanism from key-value stores — and note the extra benefit here: varying virtual node count by machine capability handles a heterogeneous fleet, the same requirement that chapter called out.

Interview signal by level

Next: what the design is actually worth, in milliseconds.