Caching and the Cost of Small Objects

In one line: this lesson contains an argument that only exists at scale — that the bookkeeping around your data can cost more than a rounding error — and it is worth checking the arithmetic, because the design overstates it.

What the cache is for

Caches reduce latency and increase throughput for storage (heavy reads), computation (stream processing), and transient data (rate limiters).

Three uses, and the third is the one people forget

The third is distinctive. A rate-limiter counter does not live in a database — the cache is the store. If it is lost, the limit resets, which is acceptable.

That changes what durability means for this tier. Losing the storage cache costs latency while it refills. Losing the transient data costs correctness, briefly, in a way nothing can repair — you simply accept it.

A cache that is the only home for its data is not a cache; it is a store you have chosen not to make durable. Worth naming, because it is the same reasoning that building block used for Redis holding driver locations: data that regenerates or expires quickly does not need durability.

The migration

Twitter historically used multi-tenant clusters of Twemcache and Redis (Nighthawk). To address operational overhead and performance inconsistency, Twitter migrated to Pelikan — a unified caching framework providing high throughput and low latency using modular backends:

• pelikan_twemcache replaces the Twemcache server.
• pelikan_slimcache replaces Memcached/Redis servers.

The stated motivation is operational, not performance

Note the reason given: "to address operational overhead and performance inconsistency."

Not "to make it faster." Two different complaints:

Operational overhead — running two different cache systems means two sets of configuration, monitoring, failure modes, deployment tooling, and on-call expertise. At thousands of instances, that duplication is expensive in engineer-time rather than machine-time.

Performance inconsistency — not slow on average, but unpredictable. Variable latency is worse than uniformly higher latency in a system with fan-out, because Lesson 7 established that a fan-out query waits for its slowest participant. One inconsistent cache node degrades every query that touches it.

So Pelikan's value is unification: one framework, modular backends, consistent behaviour.

At sufficient scale, operational uniformity is a performance feature, because variance compounds through fan-out and because engineer attention is the scarcest resource. That is a different argument from "our cache was too slow," and it is the more common real-world reason for this kind of migration.

Segcache and the metadata argument

Segcache is designed for high scalability and memory efficiency when storing small objects. In large-scale caching workloads, small objects often have median sizes in the 200 to 300-byte range. Systems such as Memcached and Redis can incur metadata overheads of roughly 50 to 60 bytes per object. For small values, this overhead can account for more than one-third of the total memory per entry. Segcache reduces per-object metadata to approximately 38 bytes.

Check the arithmetic — 'more than one-third' does not hold at the stated sizes

Two figures are given: a median object of 200–300 bytes and an overhead of 50–60 bytes. Work out the fraction:

At the stated 200–300 byte median, metadata is 14% to 23% of an entry — not "more than one-third." To exceed a third you need objects around 100 bytes, below the range the same sentence gives.

So the two claims in one paragraph are inconsistent. The likely explanation is that "more than one-third" describes overhead as a fraction of the value rather than of the entry (60/200 = 30%, still not a third), or that it comes from a workload with smaller objects than the stated median.

The saving is real and worth having, just smaller than advertised:

200-byte object:  60 B -> 38 B overhead
Entry size:       260 B -> 238 B
Memory saved:     8.5%

An 8.5% reduction in cache memory is genuinely valuable at Twitter's footprint — it is a meaningful fraction of a large hardware bill, and Lesson 3 established the cacheable working set is measured in terabytes. But it is not the transformative number the framing suggests.

Check the fraction before repeating it. An interviewer who knows this material will notice, and "the saving is about 8% at their stated object size, which is still worth millions at their scale" is a far stronger answer than reciting a third.

Why per-object metadata is a first-order cost here and nowhere else

Set the arithmetic aside — the underlying observation is genuinely important and specific to this system.

A cache entry is not just a value. It carries a key, a hash-table pointer, an expiry timestamp, LRU list pointers, a reference count, flags, and allocator padding. That bookkeeping is roughly constant per object, regardless of value size.

Which means the overhead fraction is entirely determined by object size:

Caching 1 MB video segments:  60 B overhead  ->  0.006%   irrelevant
Caching 250 B tweets:         60 B overhead  ->  19%      material

Lesson 4 explained why the objects are small: a tweet is capped at 280 characters. Lesson 3 put the text payload at 250 bytes. So Twitter is caching hundreds of millions of very small objects, which is exactly the regime where constant per-object cost stops being negligible.

The mechanism Segcache uses follows directly: amortize metadata across a segment of objects rather than storing it per object. Group objects with similar expiry times into a segment, keep expiry and eviction bookkeeping once per segment, and expire the whole segment together.

When you have many small objects, move per-object bookkeeping to per-group bookkeeping. That is the same instinct as that building block's quadtree — where structural overhead vanished because each leaf held 500 places — and the same as a B-tree using pages rather than binary nodes.

Fixed overhead becomes negligible when amortized across enough items. The design question is always what to amortize it across.

Small objects also make eviction policy matter more

A second consequence the design does not mention.

With small objects, a cache holds far more entries for the same memory — hundreds of millions rather than thousands. That changes eviction:

LRU bookkeeping itself becomes expensive. Maintaining a strict recency order across hundreds of millions of entries means pointer updates on every access, and those pointers are part of the per-object overhead being minimized.

Approximate eviction becomes attractive. Sampling a few entries and evicting the oldest is far cheaper than exact LRU and nearly as effective — the same admissible-approximation reasoning as that building block's ANN search.

Expiry-based grouping fits the access pattern. Twitter's cached objects have strongly time-correlated usefulness — recent tweets are hot, old ones are not — so grouping by expiry matches how the data actually ages.

The eviction policy should exploit whatever structure the workload has. Here the structure is time, which is the same reason Lesson 7's search index partitions by recency.

The unintuitive part of caching at this scale: when objects are small and numerous, the bookkeeping is comparable in size to the payload. A general-purpose allocator's padding and per-entry headers stop being a rounding error, which is why purpose-built cache servers manage memory in fixed-size slabs rather than deferring to malloc.

Next: observing thousands of services.