Vertical Sharding of MySQL

In one line: almost every design chapter reaches for horizontal sharding. This one does the opposite, and the reason is precise — the problem was skew across tables, not volume within one.

Two directions of partitioning

We apply vertical partitioning in two ways:

1. We split tables of a single database into multiple partitions — Partitions 2 and 3, which embed Tables 4 and 3 respectively.
2. We combine multiple tables into a single partition, where join operations are anticipated — Partition 1, which embeds Tables 1 and 2.

Therefore, we can co-locate related data and reduce traffic on hot data.

Original MySQL database
+---------+---------+---------+---------+
| Table 1 | Table 2 | Table 3 | Table 4 |
+---------+---------+---------+---------+
              |
              v
Partition 1          Partition 2      Partition 3
+---------+          +---------+      +---------+
| Table 1 |          | Table 3 |      | Table 4 |
| Table 2 |          +---------+      +---------+
+---------+
 joined together      separated because they are hot

Each partition: 1 primary + N replicas
ZooKeeper holds the mappings

Splitting and combining at the same time — and both serve the same goal

It looks contradictory: some tables are pulled apart, others pushed together. Both moves serve one objective.

Splitting hot tables apart addresses Lesson 6's skew. If questions, answers, and votes all live on one instance, that instance takes most of the QPS while others idle. Separating them means each hot table gets its own machine's worth of capacity.

Combining tables that join avoids the worst consequence of sharding. A join across partitions is not a join at all — the application fetches from both and joins in memory, losing the query planner and indexes. Keeping frequently-joined tables together means those queries stay single-partition and remain real SQL.

So the rule is: separate by heat, co-locate by join. Two criteria, sometimes in tension, and resolving that tension is the actual design work.

This is the same principle as object storage partitioning by full path and the Pub-Sub chapter choosing partition keys: partition so the dominant query touches one partition. Here the dominant queries are joins between related tables, so those tables must not be split.

Quora's problem is skew, so the fix is vertical. YouTube's was volume, so the fix was horizontal. Same database, different bottleneck, different fix — and the diagnosis is the whole question.

Vertical versus horizontal — different problems, and the diagnosis decides

Worth being precise, because these are often conflated:

Horizontal sharding splits rows of one table across machines — users 1–1M here, 1M–2M there. It solves one table being too large or too busy. That is what Vitess did for YouTube.

Vertical sharding splits tables across machines — questions here, votes there. It solves different tables competing on one instance.

Lesson 6 diagnosed Quora's problem as "a small subset of MySQL tables receives a disproportionate share of user queries" — that is skew across tables, so vertical partitioning is the matching fix.

The design is honest about the limit:

Note: vertical sharding scales by isolating tables and eliminating cross-shard joins. However, if a vertically sharded partition grows too large, horizontal sharding may eventually become necessary.

And Lesson 11 repeats it: "large tables may eventually cause write bottlenecks, requiring horizontal sharding."

So these are sequential, not alternative. Vertical first, because it is simpler and preserves joins. Horizontal later, when a single table outgrows a machine. Diagnose which bottleneck you have before choosing — using horizontal sharding for a skew problem adds complexity without addressing the cause.

The metadata

After partitioning, we require two types of mappings:

1. Which partitions contain which tables and columns?
2. Which hosts are primary and replicas of a particular partition?

Both mappings are maintained by a service like ZooKeeper.

Two mappings, and they change on completely different timescales

Worth separating, because they have different failure characteristics.

Table-to-partition changes rarely — only when you repartition, which is a planned operation. It is essentially static configuration.

Primary-and-replica hosts changes whenever a primary fails and a replica is promoted. That is unplanned, urgent, and must propagate to every client quickly — a client writing to a demoted primary is a correctness problem.

That second requirement is why ZooKeeper rather than a config file. Lesson 11 says it directly: "ZooKeeper enables service hosts to get updates about MySQL shards." Clients need to learn about failover on machine timescales, and distributed caching made exactly this argument for its configuration service — config files need a human to notice a failure, and these failures happen faster than humans.

Note this also makes ZooKeeper a critical dependency. If it is unavailable, clients cannot learn about failovers, and new clients cannot find partitions at all. Same conclusion as every coordinator in this course: it needs its own replication and consensus, which is precisely what ZooKeeper is built to provide.

Two mappings live here and they change on very different timescales. Table to shard changes rarely — only when you rebalance. Shard to host changes whenever a machine is replaced or a replica is promoted. Keeping both in a coordination service means the application never hardcodes either.

What it buys

This sharded design improves scalability by colocating related data within a single partition, avoiding cross-shard queries. Read replicas can be added for hot shards, and additional partitioning can be introduced as load increases. In scenarios that require joins across partitions, the aggregation logic can be handled at the application layer.

'Aggregation logic at the application layer' is the escape hatch, and it is expensive

Cross-partition joins are not impossible — they move to the application, which fetches from each partition and combines the results.

That is a real capability and a genuine cost, and the YouTube chapter enumerated it: you lose the query planner, you lose index-assisted joins, and you do the work in application memory. A join that MySQL would execute efficiently becomes two round trips plus a manual merge.

So this is a fallback for rare queries, not a general strategy. If cross-partition joins are common, the partitioning is wrong — you split tables that should have been co-located, and the fix is to repartition rather than to write more application-layer aggregation.

The health check worth stating: count how many of your queries cross partitions. If it is a handful of reporting queries, fine. If it is your main read path, the partition boundaries are in the wrong place.

Read replicas per partition handle skew that partitioning alone cannot

"Read replicas can be added for hot shards" is a second, finer lever.

Vertical partitioning separates hot tables from each other. But one table can still be hot on its own — the answers table is going to take more read traffic than anything else no matter how you partition.

Replicas address that: a hot partition gets more replicas, spreading reads across them while writes still go to the single primary.

So there are three levers, applied in order of increasing effort:

1. Vertical partitioning — separate hot tables from each other.
2. Read replicas — scale reads within a hot partition.
3. Horizontal sharding — split a single table when even replicas are insufficient.

Given Lesson 2's read-dominated workload — 69,444 views per second against a much smaller write rate — read replicas do a great deal of work here, and are much cheaper than horizontal sharding.

Interview signal by level

Next: replacing HBase and offloading background work.