Tiered Load Balancer Deployment

Why this matters: this is the answer to "who balances the load balancers?" In a large data center a single LB is insufficient, and the layered structure that replaces it is what candidates almost never draw.

The hierarchy

Tier 0 and tier 1

If DNS is tier 0, then equal cost multipath (ECMP) routers act as tier-1 load balancers. This layer divides incoming traffic and distributes it across multiple paths — the tier-2 LBs — using IP hashing or round robin, which is what enables horizontal scalability of everything above it.

ECMP routers are critical for scaling higher-tier LBs. Without them, you would be back to one machine receiving all traffic.

Tier 2

Tier-2 LBs are layer 4 load balancers. Their job is to ensure that all packets for a specific connection reach the same tier-3 LB, typically using consistent hashing — the mechanism from Lesson 5, applied here to LB selection rather than server selection.

They act as a stable glue layer between the raw routing of tier 1 and the intelligent processing of tier 3. Excluding tier 2 could lead to routing errors during failures or dynamic scaling, because ECMP alone gives no guarantee that a connection's packets keep landing on the same downstream device.

Tier 3

Tier-3 LBs are layer 7 load balancers. They communicate directly with backend servers and perform HTTP-level health monitoring. They also offload:

• TLS termination
• TCP congestion control
• Path MTU discovery

That offloading is the point: it lets application servers focus solely on business logic instead of network-level concerns.

Tracing a real request

Follow two requests through the tiers.

Step by step:

1. R1 arrives at a tier-1 ECMP router.
2. The ECMP router forwards R1 to a tier-2 LB via round robin. The tier-2 LB hashes the source IP to select a specific tier-3 LB.
3. The tier-3 LB terminates TLS and inspects the HTTP request. Seeing the URL path slides, it forwards to the appropriate backend server.

R2 follows the identical path through tiers 1 and 2 — but the tier-3 LB detects document in the URL and routes it to a different backend server. This content-based routing is characteristic of layer 7 load balancers such as HAProxy.

A minimal HAProxy configuration expressing exactly that:

mode http                                  # the mode the LB works in; TCP for tier-2 LBs
acl slidesApp path_end -i /presentation    # match requests whose path ends in /presentation
use_backend slidesServers if slidesApp     # send those to the slidesServers pool
backend slidesServers                      # the pool serving slidesApp
  server slides1 192.168.12.1:80           # a server in that pool

Note the comment on the first line: the same product runs in TCP mode for a tier-2 role and HTTP mode for tier 3. The tiers are as much a configuration choice as a hardware one.

Why the tier-3 fleet is larger

Look again at the diagram: there are more tier-3 LBs than tier-2 LBs. That is deliberate.

Tier 3 performs application-specific analysis and substantially more sophisticated computation, so handling the same query volume requires more machines than tier 2 needs. Tier-3 LBs also maintain the state of a large number of application servers, which would not be possible with the same machine count as tier 2.

The general shape: as you move up the tiers, per-request work increases and the fleet gets wider. Tier 1 does almost nothing per packet and is tiny; tier 3 parses every request and is the widest.

Direct server return

A natural question: should the response travel back through every tier?

No. The server can send the response directly to the routers (tier-1 LBs) through tier-3, which forwards it out of the data center. This is called direct routing (DR) or direct server return (DSR).

The reason it matters: responses are typically much larger than requests, so routing them back through every tier would make the tier-2 layer carry the full egress volume for no benefit.

Interview signal by level

Next: what you actually buy or run to get all this.