Sharding and the Cache Client

In one line: sharding removes the single point of failure, and the cache client is where the routing decision actually lives. Getting the client's invariants right is what keeps a distributed cache correct.

Dedicated cache servers

The cache servers are separated from the application servers.

Advantages:

• Flexibility to choose different hardware for application and cache needs.
• Ability to scale the application and cache tiers independently.

It also enables a "Cache as a Service" architecture, where multiple services share one cache cluster. That requires key namespacing to prevent data collisions between different applications — two services both caching key user:42 must not overwrite each other.

Co-located cache

The cache runs on the same servers as the application.

The deciding question is whether the two tiers scale together

Co-location is appealing because capacity grows for free — add an app server, get cache with it. That is only correct if the two genuinely need to grow at the same ratio.

They usually do not. Traffic growth adds application load; working-set growth adds cache pressure, and those move independently. A service whose traffic doubles but whose hot data is unchanged needs more CPU and no more cache — and co-location forces you to buy both.

Dedicated servers separate the two, which is the same "scale on your own bottleneck" argument load balancing made for tiering.

The cache client

A cache client is a library integrated into the application servers. Its job is to handle the logic for storing and retrieving data from the correct cache server, which includes performing hash calculations.

The critical invariant:

All clients must use the same hashing algorithm and server list. This ensures that a PUT for a given key is directed to the same server as a GET for that key.

Divergent client views silently break the cache

If two clients disagree about the server list — one has been updated, one has not — they hash the same key to different servers. The write lands on server A; the read goes to server B and misses.

Nothing errors. The cache simply stops working for the affected keys, quietly, with the only symptom being a degraded hit rate and increased database load. That is far harder to diagnose than a crash, and it is precisely why the next lesson introduces a configuration service to keep every client's view consistent.

Key characteristics:

• Each client knows the address of all available cache servers.
• Clients use standard network protocols like TCP or UDP to communicate.

Cache clients may also coordinate with other components, like monitoring and configuration services.

When a cache server dies

What do clients do if one of the cache servers is dead?

As the data within that cache server is no longer available, cache clients mark the access request as a cache miss.

Treating failure as a miss is the right default

It is worth appreciating how clean this is. The cache is not the design of truth, so a dead server does not mean lost data — it means the data must be fetched from the database, which is exactly what a miss already does.

There is no failover to design, no consistency question to resolve, no error to surface to the user. The failure degrades performance, not correctness. That property is a direct consequence of the volatility trade from Lesson 1, and it is what makes cache failure so much easier to handle than database failure.

The caveat: if a large shard dies, its entire key range becomes misses at once, and that load lands on the database in a burst. Availability of the cache still matters — just for capacity reasons rather than correctness ones.

Interview signal by level

Next: pinning down what we're building.