Implementation: Hardware, Software, Cloud — and Security

In one line: everything so far described what a load balancer does. This is what you actually buy, run, or rent — and the choice shapes cost, flexibility, and how quickly you can change behavior.

The three implementations

Hardware LBs offer high performance and handle many concurrent users, but they are expensive and difficult to configure. High availability is costly because it requires redundant hardware purely for failover, and they often bring vendor lock-in and limited flexibility.

Software LBs run on commodity hardware, which makes them flexible, programmable, and cost-effective. They scale easily as requirements grow, and high availability is far cheaper — shadow load balancers can be deployed on ordinary machines at minimal cost. They also support predictive analysis for traffic planning.

Cloud LBs are consumed as Load Balancers as a Service, priced by usage or SLA. They may not entirely replace on-premises solutions, but they excel at global traffic management (GSLB), and their key benefits are ease of use, metered cost, and built-in auditing and monitoring.

Why the industry moved

The economics inverted. Hardware appliances made sense when a single box outperformed anything you could assemble — you bought one very fast, very expensive machine and paid again for its idle twin.

Once commodity servers got fast enough, the software approach won on every axis that matters at scale: redundancy becomes cheap (add another commodity box), behavior becomes programmable (deploy a config change rather than file a vendor ticket), and cost scales linearly instead of in appliance-sized steps.

This is the same argument as vertical versus horizontal scaling from the Foundations module, applied to the load balancer itself.

Security at the load balancer

The LB is the first point of contact after the firewall, which makes it the natural place to enforce policy. It mitigates attacks such as denial-of-service at OSI layers 3, 4, and 7.

Layer 7 is the interesting one, because a genuinely expensive request looks identical to a legitimate one at layers 3 and 4. Only something that reads the request can tell that a client is hammering your most expensive search endpoint — which is another reason the tier-3 L7 layer exists.

What the load balancer does when everything is overloaded

Health checks handle a broken backend. The harder case is a backend that is merely overwhelmed — still answering, just slowly — because the natural responses make it worse.

Retries multiply

A retry is the standard answer to a transient failure, and it is safe exactly once. Put retries at several layers and they compose multiplicatively.

That is retry amplification, and the timing is what makes it lethal: retries are triggered by failures, so the load multiplier appears exactly when there is least capacity to absorb it. A backend at 90% that starts timing out gets 27x the requests, which guarantees it never recovers. The retry mechanism becomes the outage.

Three controls, and they are complementary rather than alternatives:

The retry budget is the one worth naming, because it inverts the usual framing. A per-request retry limit still scales with traffic; a budget is absolute, so the harder things fail the fewer retries the system permits.

The overloaded server needs help you can only give upstream

When arrival rate exceeds capacity, a server has two options and both are bad on their own.

Queueing is the subtler failure. A queue converts an overload into latency, and past the client's timeout that latency is pure waste: the server is spending its scarce capacity computing answers nobody will read, which makes the backlog grow faster.

Load shedding is the deliberate alternative: when the system is beyond capacity, reject the excess immediately rather than accepting work it cannot complete.

Two properties make it work. A fast rejection is cheap, so a shedding server can reject enormous volumes while still serving its sustainable load — where a queueing server spends the same capacity on work that will be discarded. And shedding by priority means the degradation is chosen rather than arbitrary: background jobs and retries go first, interactive requests last.

The framing worth carrying into any capacity discussion: the alternative to shedding load is not serving it — it is failing slowly instead of quickly, for everyone. A system that refuses to shed doesn't avoid the failure; it distributes it across every user and delays it until recovery is much harder.

The client-side alternative

Everything so far describes server-side load balancing: the client sends to one address and something in the middle chooses a server.

Client-side load balancing inverts that. The client obtains the list of available instances — usually from a service registry — and selects an instance directly. This is common in microservice architectures.

The modern compromise is the service mesh sidecar: a proxy deployed alongside each service instance that does client-side balancing without embedding the logic in application code. You get the direct routing and per-client policy without needing a library in every language.

Conclusion

Load balancers have evolved from hardware appliances to flexible cloud services. A robust LB layer handles traffic distribution, health monitoring, session maintenance, and security — including TLS offloading — ensuring scalability and high availability for enterprise applications.

Interview signal by level

Next: designing the whole thing under interview conditions.