The Heavy Hitters Problem

In one line: you have met the hot-key problem in four earlier chapters and each time the answer was "split the key." This chapter is that answer, worked out properly — and it is the last building block before full system designs.

The shape of the problem

A post from an account with millions of followers can receive a very high number of likes immediately after publication.

Twitter processes approximately 6,000 tweets per second, totaling about 500 million tweets daily, and the system must handle billions of "likes" generated by these tweets.

Check the numbers — they hold, and it's worth doing

6,000 tweets/second x 86,400 seconds = 518,400,000 ~ 500 million/day

Internally consistent, so you can quote either figure.

But notice which number matters here, and it is neither of those. 500 million tweets per day spread across the fleet is unremarkable — that is only ~6,000 writes/second, and any competent database handles it.

The problem is distribution. Those 500 million tweets are not equally popular: a handful receive millions of likes each within minutes, while the vast majority receive a handful. The aggregate load is easy; the load on one key is not.

That distinction is the whole chapter. Sharded counters do not increase total throughput — they redistribute it away from a single point.

The number that matters is neither the daily total nor the average rate — it is how unevenly the load lands. A design sized on averages is correct about the fleet and wrong about the one row everything is hitting.

Why writes are the problem, not reads

Write operations require exclusive access, creating bottlenecks that reads do not. As concurrent writes to a specific counter increase, lock contention grows non-linearly. Eventually, the system spends more time acquiring locks than updating the counter.

'Non-linearly' is the word to notice — this gets worse than proportionally

Naively you would expect twice the writers to mean twice the wait. It is worse than that, and understanding why explains the whole design.

Each writer must acquire the lock, update, release. As contention rises, three costs compound:

• Writers block and are descheduled, so each contended acquisition costs a context switch — pure overhead, no work done.
• The cache line holding the counter ping-pongs between cores, so even the update itself gets slower.
• Waiting writers must be woken when the lock frees, and that wake-up storm scales with the number of waiters.

The result is that past some threshold, adding writers reduces total throughput. The source states the endpoint exactly: "the system spends more time acquiring locks than updating the counter."

This is the identical failure message-queue design described for single-server queues — "the queue is a critical section, so adding clients adds contention rather than throughput" — and rate limiting noted for counter locks. Any single mutable value under concurrent writes degrades inversely with load.

Reads escape because they are not exclusive: any number can proceed simultaneously. That asymmetry is why the design that follows optimizes writes and pays for it on reads.

You have seen this problem four times already

Sharded counters are the general solution to something recurring throughout this course:

• Distributed Cache — a single hot key cannot be sharded, so replicas give a fixed 3x and beyond that you split the key into key:1, key:2, key:3.
• Distributed Messaging Queue — a hot partition key concentrates traffic on one partition; the fix is splitting the key in application code.
• Rate Limiter — a hot counter under high concurrency needs sharded counters to distribute write contention, and the chapter explicitly forward-referenced here.
• Pub-Sub — a skewed partition key produces a hot partition that nothing rebalances.

Every one is the same shape: one key, too much write traffic, no way to spread it without changing the key. Recognizing this as a single recurring problem with a single family of solutions is worth more than the specific mechanism.

Why not just make the counter faster?

The obvious alternatives all fail, and knowing why is useful in an interview:

Atomic increment instead of a lock — this genuinely helps, and rate limiting recommended it. Redis INCR removes the read-modify-write window. But it does not remove contention on the value itself: every increment still serializes on one memory location or one database row. Atomicity fixes correctness, not throughput.

A faster machine — the bottleneck is serialization, not compute. A machine ten times faster still processes those writes one at a time.

Batch the increments — real, and it is essentially what shards do at a coarser grain. But batching in one place still funnels through one place.

Approximate counting — probabilistic structures like HyperLogLog reduce space, not write contention, and they answer "how many distinct" rather than "how many total."

The only thing that removes serialization is having more than one place to write, which is precisely what sharding provides.

Interview signal by level

Next: the fix.