Why this matters: cache libraries are magnets for follow-up requirements — real ones from real operators. Interviewers pull from the same list. This lesson walks the variations worth having thought through before the round, each tested against the seams from lesson 02: does the change land in one place, or does it smear?

Capacity in bytes: the question of who knows the size

Our v1 counts entries. The day someone caches values of wildly different sizes — a 100-byte profile next to a 2 MB report — an entry count stops protecting memory, and the request becomes "cap it at 64 MB."

The design question underneath: who knows how big a value is? Inside the library, honestly, nobody. Guessing an arbitrary object's footprint from within is unreliable and invasive. The clean shape is a caller-supplied sizing function — the caller knows what they're storing:

weigher(key, value) -> cost            // caller-provided
capacity              becomes a cost budget, not a count
eviction              may need to evict SEVERAL entries
                      to admit one large one

Notice the second-order effect: eviction changes from "remove one victim" to "remove victims until the newcomer fits" — a loop where there was a step. The policy seam and the lockstep invariant survive untouched; the accounting layer changes. Being able to say which parts move is the point of having designed seams at all.

Stats: what the hit-rate is actually for

The hit-rate isn't decoration — it's the number that justifies the cache's existence. A cache holding steady at a 12% hit rate is mostly overhead with a memory bill; one at 95% is carrying a database. Two design notes worth voicing:

• Counters are contracts. Lesson 02 defined hit and miss precisely so this number means something. If updates counted as hits, a write-heavy workload would report a flattering rate while serving no reads.
• Evictions are the third signal. A high hit-rate with a high eviction rate says the working set barely fits — capacity is the lever. High misses with low evictions says the workload just isn't cacheable. One extra counter turns "the cache feels slow" into a diagnosis.

Concurrency: what sharing does to an ordering

Our stated assumption was single-threaded. Lift it, and the interesting tension appears immediately: reads are writes here. Every get mutates the recency list, so a "read-mostly" workload is a write-heavy workload for the bookkeeping, and slapping a reader-writer lock on it buys nothing.

The honest options, cheapest first:

one coarse lock        every operation serializes; simple, correct,
                       and fine until profiling says otherwise
sharded segments       hash keys into N independent sub-caches,
                       each with its own lock, map, and ordering

Sharding is the standard escape, and it comes with an admission worth making unprompted: there is no longer a single global recency ordering. Each segment evicts its own locally least-recent entry, so the globally coldest entry may survive while a warmer one in a crowded segment dies. That's an approximation you accept in exchange for parallelism — and saying so, with the trade named, reads as experience. Claiming sharding is free reads as the opposite.

Cache-aside or read-through: a library-boundary question

Our API is a pure store: on a miss the caller fetches from the source and puts the value in — the cache-aside pattern, with the loading logic living in caller code. The alternative is read-through: hand the library a loader function at construction, and get becomes get-or-load, with the cache invoking the loader on a miss.

This is genuinely a boundary decision, not a feature toggle. Read-through centralizes the load path (every caller gets the same behavior, and the library gains a place to prevent duplicate concurrent loads of the same key), but it pulls failure modes into the cache — now a get can throw a database error, and the API's promises change. Cache-aside keeps the library small, honest, and synchronous at the cost of every caller writing the same five lines. Either is defensible; what's graded is knowing they are different contracts, not different conveniences.