Evaluation: Scalability, Availability, Consistency, Cost
In one line: the remaining four requirements are where the honest limitations live — including one, cross-data-center consistency, that the design deliberately does not solve.
Scalability
Sharding distributes load as traffic patterns change, and consistent hashing minimizes the data that must be rehashed when new servers are added. Replica nodes help distribute read load across hot shards.
In rare cases, a single hot key can create contention. Mitigations:
| Mitigation | How it works |
|---|---|
| Further sharding within the key range | Split the hot key's space so the load spreads across more servers |
| Client-side dynamic replication | Clients replicate the hot key to additional servers and spread reads across them |
Why a single hot key is genuinely hard
Sharding splits key ranges. A single key cannot be split — it is the atom of the partitioning scheme. So no amount of resharding helps: the key lives on one shard, and that shard absorbs all of its traffic.
Replicas help, because reads can spread across primary and secondaries — but that is a fixed multiple (typically 3x), not a scalable one. Beyond that you must artificially split the key itself, storing key:1, key:2, key:3 and having the client pick one, then recombine.
That is the same technique the databases material suggested for hot counters, and it is real work in application code. Recognizing that this is a genuinely hard case, rather than one more sharding problem, is the senior read.
Hot writes are a different problem from hot reads
Replication spreads a hot read across replicas. It does nothing for a hot write, because every replica has to apply the write anyway.
Write batching collects updates over a short window and applies them as one operation — ten thousand increments in a second become one write of the aggregate. The trade is a small staleness window, and the precondition is that only the final value matters, which holds for counters and metrics and fails for anything where each individual event must be observable.
High availability
Redundant cache servers and leader-follower replication improve reliability and fault tolerance. But:
If all replicas reside in a single data center, the system remains vulnerable to a data center failure.
Distributing replicas across multiple data centers improves availability but introduces consistency challenges. Synchronous writes across data centers introduce high latency, so caching systems typically rely on asynchronous replication for cross-data-center deployments.
We can achieve strong consistency within a single data center, but we often compromise consistency across data centers to improve availability.
This is CAP and PACELC, stated by the design itself
The source explicitly points at CAP and PACELC here, and the mapping is exact.
CAP: during a partition between data centers, you choose availability (keep serving, accept divergence) or consistency (refuse). Caches choose availability — which is easy to justify, since a stale cache entry is recoverable and an unavailable cache dumps 20x load on the database.
PACELC's else branch is the one doing the work here though: even with no partition, synchronous cross-DC writes cost too much latency, so you choose latency over consistency all the time, not just during failures.
That is why the qualifier in Lesson 8 mattered — synchronous replication is fine "when nodes are located in close proximity" and untenable when they are not.
Consistency
Caching systems favor asynchronous writes for performance, which results in eventual consistency. Synchronous writes ensure strong consistency but increase latency.
Inconsistencies also arise during failures. For example, a server recovering after a write operation might hold stale data. To prevent this:
A rejoining server should not serve requests until it is fully synchronized.
The rejoining-server rule is the subtle one
A node that was down missed every write during its absence. If it rejoins and immediately starts serving, it answers with pre-outage data — and because the cache returns hits confidently, callers have no way to know the answer is stale.
That is worse than the node staying down. A dead node produces misses, which are correct-if-slow. A stale node produces hits, which are fast and wrong.
The rule — do not serve until synchronized — is the same fencing instinct from key-value stores's leader failover: a returning node must prove it is current before it is trusted.
Concurrency inside a shard
A subtle problem worth knowing. Many concurrent requests hit one shard: some are hits (readers), some are misses that write a new entry. These concurrent requests compete, so locking is required — but locking the entire data structure for all reads and writes heavily reduces performance.
Three alternatives:
| Approach | How it works |
|---|---|
| Limited locking | Lock only specific sections of the data structure, so some threads read simultaneously while others block on particular regions |
| Offline eviction | Record the required changes rather than applying them, committing only when necessary. Desirable when the hit rate is high, since structural changes are mostly needed on misses |
| Lock-free implementation | Proposed solutions allow simultaneous reading and writing over a doubly linked list, supporting high concurrency |
Offline eviction is the clever one
Note why it works: on a cache hit, LRU wants to move the accessed node to the head — a structural mutation requiring a lock, for what was supposed to be a pure read.
Offline eviction defers those reorderings and applies them in a batch. With a high hit rate, the overwhelming majority of operations become genuinely read-only, and contention collapses.
It is a nice example of noticing that a "read" was secretly a write, and fixing the mismatch rather than optimizing the lock.
Affordability
The design is cost-effective because it uses commodity hardware.
When the leader fails
Two options: elect a new leader among the available followers via a leader-election algorithm, or use a separate distributed configuration management service to monitor and select leaders.
The second reuses the configuration service from Lesson 8 — one component solving both server discovery and leader selection.
Key takeaway
Strong consistency within a data center, eventual consistency across them — a deliberate PACELC choice driven by latency, not just partitions. The remaining hard cases are single hot keys and intra-shard concurrency, both of which need application-level work rather than more servers.
Interview signal by level
| Level | What a strong answer sounds like |
|---|---|
| L4 | "Replicas make it available and sharding makes it scale." |
| L5 | Names the cross-DC trade: "replicas in one data center give strong consistency but don't survive losing it — across data centers we'd go async and accept eventual consistency." |
| Staff+ | Reaches PACELC and the hard cases: "it's the else branch that decides this — sync cross-DC writes are too slow even with no partition, so we choose latency permanently. And a single hot key can't be sharded, so replicas give a fixed 3x and beyond that you split the key in application code. A rejoining node also mustn't serve until synced, or it returns confident stale hits." |
Next: how the real implementations differ.