Requirements, API, and Design Considerations
In one line: the API here is two functions and looks identical to a key-value store's. Being able to say precisely how the two differ is one of the sharpest questions in this chapter.
Requirements
Functional — the system must support:
- Insert data: users must be able to add an entry to the cache.
- Retrieve data: users must be able to retrieve data associated with a specific key.
Non-functional:
| Requirement | Detail |
|---|---|
| High performance | The primary goal — both insert and retrieve must be low-latency |
| Scalability | Scale horizontally to handle increasing request loads without bottlenecks |
| High availability | Cache downtime increases load on the database, potentially causing cascading failures during peak traffic. Must survive component failures, network issues, and power outages |
| Consistency | Data consistent across cache servers — clients reading the same key from different servers (primary or secondary) should see up-to-date data |
| Affordability | Use commodity hardware to minimize costs |
Read the availability requirement carefully — it's the non-obvious one
"Cache downtime increases load on the database, potentially causing cascading failures during peak traffic."
The cache is not the design of truth, so losing it costs no data. But losing it redirects the entire read load onto a database that was sized assuming the cache absorbs most of it. A cache with a 95% hit rate that goes down does not increase database load by 5% — it increases it by 20x.
That is why cache availability matters despite the cache being expendable. It is a capacity dependency, not a correctness one, and it is the mechanism by which a cache outage becomes a total outage.
API
insert(key, value)
| Parameter | Description |
|---|---|
key | A unique identifier |
value | The data stored against a unique key |
Returns an acknowledgment or an error indicating the status of the operation.
retrieve(key)
| Parameter | Description |
|---|---|
key | Returns the data stored against the key |
Returns the requested object to the caller.
Cache versus key-value store
The API looks exactly like a key-value store's. The differences are real and worth being precise about:
| Key-value store | Distributed cache | |
|---|---|---|
| Durability | Must durably store data — it is the source of truth | Used in addition to persistent storage, to increase read performance |
| Where data lives | Non-volatile storage | Served from RAM |
| Failure expectation | Robust — should survive failures | Can crash and be populated from scratch after recovery |
Same interface, opposite guarantees
Two systems can share an API and be fundamentally different, because the API describes operations while the guarantees describe obligations.
A key-value store that loses data has failed. A cache that loses data has had a bad afternoon. That single difference is why key-value stores needed vector clocks, hinted handoff, and Merkle-tree anti-entropy — all machinery for not losing writes — and why none of it appears here.
If an interviewer asks how the two differ, lead with durability obligation, not with the feature list.
The APIs look identical — get, set, delete — which is exactly why the distinction gets lost. The difference is what it means to lose the data, and that single question decides whether eviction is a feature or a bug.
Design considerations
Storage hardware
Large datasets require sharding across multiple servers. Specialized hardware offers better performance but costs more; alternatively, build a large cache from commodity servers. The number of shard servers depends on cache size and access frequency.
You can persist data to secondary storage while serving from RAM. Persistence aids recovery if a reboot occurs, as rebuilding the cache from scratch takes time — but may not be necessary if a dedicated persistence layer already exists.
Data structures
Hash tables provide constant-time O(1) average complexity for storage and retrieval, and linked lists track entry order for eviction — the pairing from Lesson 4.
The design uses strings for simplicity, but caches can store hash maps, arrays, or sets.
Cache client
The client library or process handles insert and retrieve. It can reside on the service host for internal use, or exist as a dedicated client for external services.
Writing policy
There is no single optimal choice; the policy depends on application requirements — Lesson 2.
Eviction policy
Cache size is limited compared to the full dataset, so a policy decides what to keep. LRU is effective for social media services where recent content attracts the most views. Additionally, optimizing the TTL value is essential for reducing cache misses.
How sharding removes the single point of failure
Sharding splits the key space so no single server holds everything. If one shard dies, only its key range becomes misses — the rest of the cache keeps serving.
Without sharding, one cache server failing means 100% of requests fall through to the database. With ten shards, one failure means 10%. The blast radius scales down with the shard count, which is the same containment argument as cells in CDNs.
Replication, added in Lesson 8, then shrinks even that 10% to near zero.
Key takeaway
Two operations, five non-functional requirements. The cache differs from a key-value store not in its interface but in its obligation — it may lose everything and still be working correctly.
Interview signal by level
| Level | What a strong answer sounds like |
|---|---|
| L4 | "Get and put, and it should be fast and scalable." |
| L5 | Distinguishes it from a KV store: "same API, but the cache isn't durable — it serves from RAM and can be rebuilt from the database after a crash." |
| Staff+ | Quantifies the availability requirement: "cache downtime doesn't lose data, it redirects load — a 95% hit rate means losing the cache multiplies database traffic by 20x. That's how a cache outage cascades into a full outage, and it's why availability matters for a component that's technically expendable." |
Next: the first architecture.