Shard Selection: Round Robin, Random, Metrics-Based

In one line: the interesting result here is that random beats round robin in the distributed case, which is the opposite of what most people assume. Understanding why is a genuinely transferable insight.

Round robin

Write requests are assigned to shards sequentially. With 100 shards, the system sends requests to shard 1, then shard 2, and continues until shard 100 before repeating.

Round robin does not account for each shard's current load, leading to some shards being overloaded while others remain underutilized. However, when requests are uniform and have similar processing costs, this approach works reasonably well and is attractive due to its simplicity.

Shard_1  Shard_2  Shard_3  Shard_4
  12       66       31       65
  BUSY     BUSY     free     BUSY
   ^                  ^
   |                  |
next write must       this shard is idle
wait for shard_1      but round robin
(cycle position)      won't use it yet

Round robin can block on a busy shard while an idle one sits unused

The slides make the failure concrete. The server assigns writes to shards 1, 2, 3, 4 in order. Shard 3 finishes and is free. But the next write is due for shard 1, which is still busy — so the write waits, "though shard_3 is done with incrementing its counter and is free, we can't utilize it because of the round-robin pattern."

That is round robin's defining weakness: it is position-driven, not availability-driven. It cycles regardless of what is actually free.

And the design names an aggravating factor specific to this architecture:

In practice, requests are first routed to servers by a load balancer. Each server then selects a shard using round robin. As shards may receive requests from multiple servers concurrently, some shards can become overloaded.

This is the crucial detail. Round robin is only fair within one server's view. With M servers each independently cycling through the same N shards, their cycles can align — several servers hitting shard 1 simultaneously, then all moving to shard 2 together. You get synchronized convoys rather than even spread.

So the mechanism that guarantees fairness locally produces correlated collisions globally, and the more servers you add, the worse it gets.

Random

Another simple approach is to select a shard uniformly at random for each write.

The limitation of both round robin and random routing is that they do not account for dynamic load variation across the nodes hosting the shards. Load can become unevenly distributed across shards. Load variability is common because each physical node typically runs multiple services and workloads.

Random is usually better than round robin here — and the reason is worth internalizing

Both are load-oblivious, so the design groups them. But in the multi-server setting, random is strictly better, and the reason is subtle:

Round robin's guarantee is local and its failure is correlated. Each server spreads perfectly across shards, but multiple servers can synchronize into convoys hitting the same shard together.

Random has no guarantee and no correlation. Any single write may land on a busy shard, but independent random choices cannot systematically align. With enough writes, the distribution is even by the law of large numbers, and there is no pathological pattern to fall into.

This is the same result as hash-based versus round-robin load balancing generally: when several independent dispatchers share a pool, randomness avoids the resonance that deterministic cycling can create.

It is also why the design's own architecture, shown in the worked slides, uses Random (Shard) as its selection step rather than round robin. The chapter presents round robin first for pedagogy and then quietly does not use it.

Metrics-based

The third approach is to select shards based on specific metrics. For example, a dedicated node (load balancer) manages the selection of the shards by reading the shards' status.

Metrics-based selection is the best answer and the most expensive one

Choosing the least-loaded shard is obviously better than choosing blindly. The costs are real:

Someone must observe every shard's load, continuously, and that observation is stale the moment it is taken — during a burst, shard load changes faster than you can measure it.

The load balancer becomes a coordination point on the write path, which is the thing sharding existed to remove. Push too much decision-making into it and you have recreated a single serialization point one level up.

Stale load data causes herding. If every dispatcher learns that shard 7 is least-loaded, they all send to shard 7 — and it becomes the most loaded. This is a classic distributed-systems failure, and the standard mitigation is power of two choices: sample two shards at random and pick the less loaded of them. It captures most of the benefit of load-awareness with almost none of the coordination cost, and it cannot herd because the samples differ per dispatcher.

If asked to choose in an interview, random for simplicity, or power-of-two-choices if load genuinely varies is a stronger answer than "metrics-based," because it shows you know what full load-awareness costs.

Why load varies across shards at all

The source gives a reason that is easy to skip: "each physical node typically runs multiple services and workloads."

That is the point. Shard load is not determined only by the writes you send it. The node hosting shard 7 might also be running a batch job, serving another tenant, or garbage collecting. So even perfectly even write distribution produces uneven completion times.

This is why load-oblivious strategies degrade in practice and why it is the same argument load balancing made for weighted round robin and distributed caching made for virtual nodes sized to machine capability: uniform distribution is only correct when the targets are uniform, and at fleet scale they never are.

Choosing

Strategy	Choose when	Cost
Round robin	Requests are uniform with similar processing costs and there is one dispatcher	Position-driven, not availability-driven; multiple servers can synchronize into convoys
Random	The default. Multiple dispatchers, uniform-ish shards	Load-oblivious, but errors are uncorrelated and even out
Metrics-based	Shard load genuinely varies and the coordination cost is affordable	Continuous observation, a coordination point on the write path, herding on stale data

The reason random wins more often than it should: the counter's value is a sum across all shards, so uneven distribution between shards does not affect correctness at all — only the load on individual shards, and that evens out at volume.

Key takeaway

Round robin is position-driven, so it can block on a busy shard while an idle one waits — and with multiple dispatchers, independent cycles synchronize into convoys. Random has no guarantee but its errors are uncorrelated, which makes it better in the multi-server case and is what the design actually uses. Metrics-based is best in principle but pays for observation, adds a coordination point, and herds on stale data — for which power of two choices is the practical middle.

Interview signal by level

Level	What a strong answer sounds like
L4	"Pick a shard at random for each write."
L5	Compares the options: "round robin is simple but ignores current load; random is similar but doesn't get stuck waiting on a specific shard; metrics-based picks the least loaded but needs monitoring."
Staff+	Explains why random beats round robin here: "round robin's fairness is local — with several servers each cycling independently through the same shards, their cycles can align into convoys hitting one shard together, and it gets worse as you add servers. Random has no guarantee but its errors are uncorrelated, so it can't resonate. Full metrics-based selection puts a coordination point back on the write path and herds when load data is stale — I'd use power of two choices instead, sampling two shards and taking the lighter, which gets most of the benefit without the coordination."

Next: making reads cheap again.