Interview Walkthrough: Design a Load Balancer

"Design a load balancer."

The functional surface is one sentence — take a request, pick a server, forward it. Everything that makes this a Staff-level problem is non-functional: it must be faster than what it fronts, more available than what it protects, and it must not drop connections while any part of it changes.

Step 0 — Scope it before you design

• Which layer? L4 or L7 — do we need to route on URL and headers, or just move connections?
• What scale? Requests per second, concurrent connections, bytes per second. These three size very different things.
• One data center or global? Do I own the GSLB tier too?
• Do we terminate TLS? This drives CPU cost enormously.
• Are backends stateless, or do we need session affinity?
• What protocols? HTTP only, or arbitrary TCP?

Then commit:

"I'll design the layer for a single large data center — I'll assume GSLB above me picks the region. L7, because we need path-based routing and TLS termination. Let's say 1 million requests per second at peak, 10 million concurrent connections, backends are stateless so I don't need affinity. Correct me on any of that."

Step 1 — A quick BOTEC

Using the method from the Foundations module:

Peak requests            = 1,000,000 RPS
Concurrent connections   = 10,000,000
Per-LB capacity (L7, TLS) = ~100,000 RPS      (TLS + parsing is expensive)

LBs by request rate      = 1M / 100K          = 10 boxes
Plus N+1 redundancy and headroom              = ~15 boxes

Memory per connection    = ~10 KB
Memory for 10M conns     = 100 GB spread over the fleet -> ~7 GB per box

"So roughly 15 L7 boxes. That's already too many for a single VIP to feed, which tells me I need something in front of them — that's the tier-1 layer. The number itself generated the architecture."

Step 2 — Functional Requirements

• Distribute incoming requests across a pool of backend servers.
• Health check backends and remove unhealthy ones from rotation.
• Add and remove backends without dropping traffic.
• Route by content (L7) — path, header, or host.
• Terminate TLS.

Step 3 — Non-Functional Requirements — the heart of this problem

Step 4 — Core Entities & Key State

Backend
  address, port
  weight            -- for weighted algorithms
  health_status     -- healthy | draining | unhealthy
  active_connections
  in_pool_since     -- drives slow start

RoutingRule (L7)
  match             -- path prefix, host, header
  target_pool

The draining state is the one worth pointing at. A backend being removed is not simply deleted — it stops receiving new connections while existing ones finish. Without that intermediate state, every deploy kills in-flight user requests.

Step 5 — API & Interfaces

Two surfaces, and conflating them is the classic error:

Data plane — there is no API. The load balancer is transparent: clients open a normal TCP connection to a VIP and speak HTTP. They do not know an LB exists, and that invisibility is the product.

Control plane — a normal authenticated management API, used rarely:

POST   /pools/{pool}/backends       (address, port, weight)
DELETE /pools/{pool}/backends/{id}  -> transitions to draining, then removed
PUT    /pools/{pool}/health-check   (path, interval, thresholds)
PUT    /routes                      (match rules -> pools)
PUT    /certificates                (TLS material)

Step 6 — High-Level Design

(a) The naive core

Works, and fails the first NFR: the LB is now a single point of failure in front of 100% of traffic, and it caps throughput at one machine.

(b) Remove the single point of failure

A pair sharing a floating virtual IP: if the active dies, the VIP moves to the standby. Good for availability, but capacity is still one machine — active-passive doubles cost without doubling throughput.

(c) Scale the LB fleet — and face the new problem

Run many LBs active-active. But now: what distributes traffic across the load balancers?

Narrate the three tiers as answers to three distinct problems:

"ECMP routers spread traffic across the LB fleet at layer 3 — that's what makes the fleet horizontally scalable. The L4 tier uses consistent hashing so every packet of a connection lands on the same L7 box, which ECMP alone can't guarantee during scaling or failure. The L7 tier does the expensive work: TLS termination, content routing, HTTP health checks. Tier 3 is the widest fleet because it does the most per request."

(d) The mature design

Add: connection draining on removal, slow start on addition, active plus passive health checking, a control plane distributing pool and certificate config, and DSR for large non-TLS responses.

Deep Dives & Follow-up Questions

"An L7 load balancer dies mid-connection. What happens?"

In-flight connections on that box are lost — it holds TLS session state, so there's no way to move them silently. Clients see a reset and retry, which for idempotent requests is invisible. What I care about more is that the failure doesn't spread: the L4 tier's consistent hashing means removing one L7 box only remaps its share of the hash ring, roughly 1/N of connections, rather than reshuffling everyone. That's exactly why tier 2 uses consistent hashing instead of modulo.

"How do you add a backend without breaking anything?"

Register it in the control plane, but bring it in with slow start — ramp its share of traffic over 30 to 60 seconds. Otherwise a least-connections algorithm sees a server with zero connections, floods it, the cold server responds slowly with empty caches, fails its health check, and gets ejected. That flap loop is caused by the balancing algorithm and health checker interacting, and slow start is what breaks it. Removal is the mirror image: mark it draining, stop new connections, let in-flight requests finish, then remove.

"One backend is slow but not down. What happens?"

With a static algorithm like round robin, nothing — it keeps receiving its full share and a fraction of every user's requests get slow. That's the temporal failure mode from Foundations, and it's why I'd use a dynamic algorithm. Least connections naturally sheds load from it, since its in-flight count stays high. On top of that, passive health checking with outlier detection ejects any backend whose latency or error rate diverges sharply from its peers — a probe alone would never catch this.

"You have 10 load balancers using least connections. Any problem?"

Yes — each one only sees its own connections, so all ten can independently decide the same backend is least loaded and send it a simultaneous burst. The balancing algorithm creates a herd. I'd use power-of-two-choices instead: pick two backends at random, send to the less loaded. It gets close to true least-connections, needs no shared state, and two LBs rarely sample the same pair, so the herding disappears.

"Do responses go back through every tier?"

They don't have to. Direct server return lets the backend send the response out through tier 3 to the routers without traversing tier 2 — which matters because responses are usually far larger than requests, so routing them back through every tier would make tier 2 carry full egress volume for nothing. The catch is TLS: if we terminate at tier 3, the response must pass back through that box to be encrypted, because that's where the key material and connection state live. So DSR is mainly for non-TLS traffic.

"Where's the bottleneck at 1M RPS with TLS?"

CPU on the L7 tier, spent on TLS handshakes and HTTP parsing. Handshakes are far more expensive than resumed connections, so session resumption and keep-alive matter enormously — a client that reconnects per request can cost an order of magnitude more than one holding a connection open. If CPU is still the limit I'd offload crypto to dedicated hardware, or push simple high-volume paths down to L4 where no parsing happens.

"How does the control plane distribute config without becoming a dependency?"

Push config to the LBs and have them run on their last known good copy. If the control plane is unreachable, the data plane keeps serving with the configuration it already has — static stability. The failure I want to avoid is a control-plane outage becoming a traffic outage, which happens when LBs treat "can't reach the registry" as "the pool is empty." Health checks are local to each LB for the same reason: no synchronous external lookup on the hot path.

Now do it live

The next section drills these as standalone probes.