Why Client-Side Load Balancing
In one line: the previous lessons all assumed a load balancer in front of everything. This one explains why Twitter removed it, and the reasoning applies to any sufficiently large microservice fleet.
The history
Twitter began as a monolithic Ruby on Rails application with a MySQL database. As the platform scaled, the database was sharded, but the monolithic application remained problematic.
| Problem | Detail |
|---|---|
| Codebase contention | A large number of developers working on the same codebase made it difficult to update individual services |
| Cascading failures | An upgrade in one service could break others |
| High costs | Single machines performing numerous services led to inefficient hardware usage |
| Complex recovery | Recovering from failures was slow and difficult |
Twitter eventually migrated to microservices, where each service runs on hundreds or thousands of instances.
Three of the four problems are organizational, not technical
Read the list again. Codebase contention is about developers. Cascading failures are about coupled release cycles. Complex recovery is about operational process.
Only high costs — one machine doing many things — is a machine problem, and it is the weakest of the four.
That is the honest reason most monoliths get broken up, and it is worth saying in an interview because it is usually misstated. Microservices are rarely adopted because a monolith cannot scale computationally; a monolith scales horizontally perfectly well behind a load balancer. They are adopted because a monolith does not let independent teams ship independently.
The costs are real and worth naming too: network calls where there were function calls, distributed failure modes where there were exceptions, and — as this chapter demonstrates — entirely new infrastructure problems like the one that follows. Twitter needed a service registry, distributed tracing (Lesson 9), and a custom load-balancing algorithm, none of which a monolith requires.
Microservices trade technical complexity for organizational independence. Whether that trade is worth it depends on how many teams you have, not how much traffic.
Why the centralized load balancer fails
The previous lesson modeled Twitter with a centralized load balancer. Although functional, this design does not scale efficiently for Twitter's traffic patterns. Twitter operates many heterogeneous services across large instance fleets, which can overwhelm centralized load balancers and create throughput bottlenecks.
Internal traffic, not external traffic, is what breaks it
The key phrase is "many heterogeneous services." This is not about users hitting Twitter — it is about services calling each other.
A single user request fans out internally: the timeline service calls the tweet store, the counter service, the user service, the ad service, each of which calls others. So internal request volume is a large multiple of external request volume — the same amplification Lesson 3 found in the write path, now in the call graph.
Route all of that through centralized load balancers and four things go wrong:
Bandwidth funnels through one machine. Every byte between two services traverses a third. At internal-traffic volumes that is a hard ceiling.
An extra network hop on every call. A → LB → B instead of A → B. Small individually; multiplied by a deep call graph, material. And it lands on the latency-critical path Lesson 2 said must stay fast.
It is a failure domain. The load balancer for a service is a single point of failure for every caller of that service.
It becomes a queue. All callers wait on one resource, so its queue is shared — one slow backend can back up traffic for everyone, which is the "users waiting on a single resource" the design calls out under quality of experience.
A centralized load balancer is a shared resource on the critical path of every internal call, and shared resources on hot paths are exactly what a scaled architecture removes.
The alternative
In client-side load balancing, request routing does not rely on a centralized load balancer. The client embeds load-balancing logic and selects an appropriate backend instance directly. Backend instances register with a service registry so clients can discover available endpoints.
Service A acts as the client for Service B. It uses an internal load balancer to select the best instance. Service B subsequently acts as a client for Services C and D. This removes the central load-balancing entity; every node manages its own traffic routing.
| Advantage | Detail |
|---|---|
| Reduced infrastructure | Fewer hardware layers are required |
| Lower latency | Removing the intermediate hop reduces network latency |
| No bandwidth bottlenecks | Traffic does not funnel through a single machine |
| Resilience | Fewer points of failure in the system |
| Improved QoE | Removing the central queue prevents users from waiting on a single resource |
The service registry is the piece that makes this work — and it is the new dependency
Removing the load balancer does not remove the need for the thing it knew: which instances exist and which are healthy.
That knowledge moves to the service registry, and every client now consults it. Which means the registry has the properties a coordination store needs, exactly as Lesson 9 described:
- Small — a list of instances, not data.
- Strongly consistent — a client must not route to a dead instance.
- Read by everyone — every client, continuously.
- Changed rarely — instances come and go on deploys and failures.
That is ZooKeeper's job, and Lesson 9 confirms Twitter uses it for exactly this: "service registries."
Be honest about what has actually changed. You have not eliminated a shared dependency; you have replaced a shared component on the data path with a shared component on the control path. The registry is consulted to learn topology, not to carry every byte, and clients cache what they learn.
Moving a dependency off the data path and onto the control path is the general form of this optimization — it turns per-request cost into per-change cost. The same instinct as DNS, and as that building block's key-value store mapping segments to servers.
What client-side load balancing costs
The design lists only advantages. The costs are real and worth volunteering.
Logic in every client. Load-balancing code is now embedded in every service, in every language you use. Twitter can do this because Finagle (Lesson 12) is a shared library and their services are largely JVM-based. A polyglot shop pays this cost repeatedly — which is precisely the problem service meshes exist to solve, moving the logic into a sidecar proxy instead.
Every client holds connections. With a central load balancer, backends see connections from a handful of balancers. Now every client connects to backends directly, and the connection count multiplies — which is exactly the problem Lesson 13 spends its entire length solving.
Distributed decisions with local information. A central balancer sees all traffic and can balance perfectly. Each client sees only its own, so decisions are made on partial information — which is why Lesson 12 needs an algorithm that works well without global knowledge.
Upgrades are deploys. Changing the balancing algorithm means redeploying every client, rather than reconfiguring one tier.
So the trade is: remove a bottleneck and a hop, gain a distributed coordination problem. The next two lessons are that problem.
At internal-service call volumes the middle hop is not free — it is latency on every request plus a tier to run. Moving the decision into the client removes both, at the cost of every client needing service discovery and a load-balancing policy of its own.
Key takeaway
Three of the four monolith problems are organizational — codebase contention, coupled releases, recovery process — which is the honest reason monoliths get broken up: microservices trade technical complexity for organizational independence, and that trade depends on team count, not traffic. The centralized load balancer fails on internal traffic, which is a large multiple of external traffic, because it funnels bandwidth, adds a hop to every call, forms a failure domain, and becomes a shared queue. Client-side balancing removes it but does not remove the shared dependency — it moves the registry from the data path to the control path, turning per-request cost into per-change cost. The costs are real: logic in every client, multiplied connections, and decisions made on partial information — which is what the next two lessons address.
Next: choosing an instance with only local knowledge.