Why a Distributed Cache

In one line: almost every design in this course puts "add a cache" on the board. This chapter is what that box actually contains once the cached data no longer fits on one server.

The problem

A typical system has three components: the client who requests the service, the service host that processes requests, and the database that stores the data.

That works at low traffic. Scaling up the number of users increases database query volume, which overloads the database and causes high latency.

Caches store only the most frequently accessed data. On a request, the serving host retrieves from the cache (cache hit) and serves the user. If the data is absent (cache miss), it is retrieved from the database — and after a miss, the cache is populated with the new value to prevent future misses.

Why RAM

A cache stores transient, frequently accessed data to reduce latency. So the storage hardware must be fast, large enough to hold the working set, and cost-effective. RAM is the standard building block for caching.

CPU register    fastest, tiniest, most expensive
CPU cache         ^
RAM               |   <-- area of interest: enough capacity, still volatile
SSD               |
Magnetic disk   slowest, largest, cheapest

decreasing cost and speed  ->
increasing capacity        ->

RAM sits at the point on that hierarchy where speed is still excellent and capacity is affordable enough to hold a working set. Everything faster is too small; everything larger is too slow.

Why distribute it

Storing the entire dataset on a single system is often impractical. Distribution addresses three limitations:

Limitation	How distribution helps
Single point of failure	Prevents total system failure if one node crashes
Modular design	Each architectural layer can have its own caching mechanism, decoupling sensitive data
Latency	Caching at different locations places data closer to the request source

Distributed caches rely on the locality of reference principle — the assumption that programs access specific data subsets repeatedly. Without that assumption a cache would be pointless; with it, a small fast tier absorbs most of the traffic.

Their key benefits: reduced latency, database offloading, session storage, availability (serving data even if the primary data store is temporarily down), and scalability through horizontal scaling.

Caching at different layers

System layer	Technology in use	Usage
Web	HTTP cache headers, web accelerators, key-value store, CDNs	Accelerate retrieval of static web content, manage sessions
Application	Local cache and key-value data store	Accelerate application-level computations and data retrieval
Database	Database cache, buffers, key-value data store	Reduce data retrieval latency and I/O load from the database

How it works, end to end

The application requests data from the distributed cache.
If the data exists, the server returns it immediately.
If missing, the cache server retrieves it from the backend database.
The new data is stored in the cache for future requests.
The data is returned to the application.

Cache servers are deployed in a cluster to improve performance and scalability, letting the web server avoid retrieving from the database for every request.

Best practices, previewed

Practice	Detail	Covered in
Cache eviction	Policies such as LRU or TTL keep the cache refreshed and relevant	Lesson 3
Data consistency	Keep cache and primary source aligned, especially for frequently updated data	Lessons 2 and 10
Monitoring	Track hit and miss rates to identify areas for improvement	Lesson 9
Scalability	Design so cache nodes can be added easily as the application grows	Lessons 5 and 10

The industry options

Redis — an open-source in-memory data structure store, known for speed and scalability.
Memcached — another popular open-source distributed cache, simple to use and easily scalable.
Hazelcast — commercial, with advanced features including data replication and eventing.
Apache Ignite — open-source distributed caching and computing, with in-memory processing and distributed SQL.

Lesson 11 compares the first two in depth.

Key takeaway

Cache the data that is frequently accessed and rarely changes, set expiration to keep it fresh, and monitor hit rates. Distribution enters the picture when the working set outgrows one machine — and brings the partitioning, replication, and consistency problems the rest of this chapter solves.

Interview signal by level

Level	What a strong answer sounds like
L4	"Add a cache in front of the database to reduce load."
L5	Says which layer and why: "an application-layer cache between the services and the database, serving from RAM so reads don't hit disk."
Staff+	Frames the volatility trade: "the cache isn't the source of truth, which is what lets it be volatile and cheap to rebuild. And it only works if there's locality of reference — if access is uniform across a huge dataset, a cache buys nothing."

Next: when to write, and where.