Placement and Storage

In one line: the storage layout is two different systems doing two different jobs — a key-value store for the mapping on the write path and a wide-column store for aggregates on the read path. Knowing why each was chosen is the point.

Placement

In social feed workloads, deploying counters closer to users can reduce write latency and improve aggregation performance for heavy-hitter and Top-K queries.

Edge placement composes perfectly with Lesson 7's regional counters

The three placements are not equally suited to this workload, and the edge option is the one that fits the design already built.

Lesson 7 established regional counters as both a product feature and a write-locality mechanism. Edge placement is the physical realization of that: a like from Tokyo increments a shard in Tokyo, never crossing an ocean.

Three things follow:

• Write latency is minimal — the increment stays within the region.
• Regional Top-K is computed locally, where the counters already live, which is exactly what Lesson 7's local-lists-then-merge structure needs.
• The global rollup is asynchronous and off the write path — it is a derived value, so nothing waits for it.

The cost is more places to aggregate from, and Lesson 6's periodic aggregation already absorbs that. The placement decision and the counter hierarchy reinforce each other, which is a good sign the design is coherent rather than assembled.

The storage layout

Reads can rely on a scalable data store to serve counter values. For example, Cassandra can store region-specific counts (views, likes, comments, etc.) as the latest aggregated sum of a counter's shards.

When users load a timeline, read requests are routed to nearby servers, which respond with these stored values.

Separately, the system needs storage for shard metadata and mappings. Redis or Memcache can store:

• tweetId → counterId(s) (like/reply/retweet, etc.)
• counterId → list of shard IDs

Two stores for two access patterns — this is the recurring 'choose by query shape' rule

The design uses Redis/Memcache and Cassandra for genuinely different jobs:

Redis holds the mappings, which are consulted on every single write. They are tiny, read constantly, changed rarely (only on create or resize), and always fetched by exact key. That is an in-memory key-value store's exact profile — and critically, the lookup sits on the write path, so it must cost microseconds.

Cassandra holds the aggregated counts — larger, written periodically by the aggregator, read by timelines, and needing regional distribution and durability. That is a wide-column store's profile, and the databases material's argument for it applies: high write throughput, tunable consistency, geographic replication.

This is the same principle as object storage splitting metadata between relational and key-value stores, and that building block splitting between relational and graph: choose the store by the shape of the access, not by the shape of the data. Third appearance, same rule.

Two lookups on every write — and that is the price of indirection

Trace a single like:

tweetId   -> counterId(s)     (Redis lookup 1)
counterId -> list of shards   (Redis lookup 2)
pick one at random
increment that shard

Two mapping lookups before any counting happens. That is real overhead on the hottest path in the system.

What buys it is flexibility. Because the shard list is data rather than a computed function, Lesson 4's dynamic resizing works: change the list in Redis and every subsequent write routes differently, with no client change and no rehashing of existing values.

Compare with distributed caching's consistent hashing, which computes placement with no lookup at all — but then constrains how you can redistribute. This design pays two lookups to keep resizing trivial, and given that Lesson 4 established shard count must change by orders of magnitude within hours, that is the right trade.

Computed placement is cheaper; directory placement is more flexible. Here, flexibility wins.

The write path

Writes can be processed in parallel: requests map to the right counter, then pick a shard (often using randomized or load-aware selection) to apply increments/decrements. Aggregation runs periodically to sum shard values, and the result is stored back in Cassandra.

The worked sequence:

1. Users send write requests to the application server.
2. The application server identifies the appropriate counters against tweets in the Redis key-value store.
3. The application server randomly chooses the shards against counters in the Redis key-value store.
4. The selected shards are highlighted and incremented.
5. The sum of the values of all shards is computed and stored back.

Note that the design actually uses random selection

Step 3 says randomly chooses the shards, and the recovered diagram labels the step Random (Shard).

Lesson 5 walked through round robin first and identified its convoy problem with multiple dispatchers. The design's own worked example quietly uses random — which is consistent with that analysis and worth pointing out.

The prose hedges with "often using randomized or load-aware selection," leaving metrics-based available. But the default here is random, and Lesson 5 explained why that is defensible: uncorrelated errors beat locally-fair-but-globally-synchronized ones.

Aggregation writes back to Cassandra, which decouples the read path entirely

Step 5 stores the computed sum back into Cassandra, and that is what makes reads cheap.

A timeline read never touches a shard. It reads one aggregated value from a nearby Cassandra replica — the same cost as reading an unsharded counter. All the read amplification from Lesson 4 has been moved off the request path into the periodic aggregator.

So the final shape is:

• Write path: two Redis lookups, one shard increment. Fast, parallel, uncontended.
• Aggregation path: periodic, off both critical paths, absorbs the fan-out cost.
• Read path: one lookup of a pre-computed value. Fast.

Sharding made writes cheap and reads expensive; periodic aggregation makes reads cheap again by paying the cost in the background. That is the complete answer to Lesson 2's asymmetry, and it only works because Lesson 6 established that exactness is unobtainable anyway.

The tension worth stating: co-locating a counter's shards makes the aggregation read cheap, and puts all of that counter's write traffic back on one partition. Spreading them fixes the write hotspot and makes every read a scatter-gather. There is no arrangement that is optimal for both.

Interview signal by level

Next: what the design achieves.