Load Balancing Algorithms
In one line: the algorithm is the load balancer's actual decision. Pick one that ignores server state and a single slow machine silently degrades a fraction of every user's requests.
The six algorithms
| Algorithm | How it decides | Reach for it when |
|---|---|---|
| Round robin | Forwards requests to servers sequentially in a repeating loop | Servers are identical and requests are uniform |
| Weighted round robin | Each node is assigned a weight; higher-weighted servers receive a larger proportion of requests | Servers have varying capacities |
| Least connections | Assigns each new request to the server with the fewest active connections | Request processing times vary significantly |
| Least response time | Directs traffic to the server with the lowest active response time | Latency is the priority — performance-sensitive services |
| IP hash | Hashes the client's IP address to map them to a specific server | You need session stickiness |
| URL hash | Distributes requests based on the URL | Different clusters serve distinct workloads |
Other variations exist, such as randomized and weighted least connections.
The two that need elaborating
Least connections matters more than it first appears. When request processing times vary significantly — some requests return in 5 ms, others run for 30 seconds — round robin keeps handing new work to a server already tied up with long-running tasks. Least connections prevents that server from becoming overwhelmed, because its high in-flight count makes it unattractive until it drains.
URL hash is about routing to specialized clusters, not balancing across identical ones. It routes traffic to specific server clusters handling distinct services — separating video serving from text, for instance. The servers behind different URL paths are genuinely different fleets with different hardware profiles.
IP hash buys stickiness, and it costs you
Hashing the client IP always maps a given client to the same server, which is how you get session affinity without shared session storage. But it creates real problems: clients behind a large corporate NAT all hash to one server, users on mobile networks change IP and lose their session, and adding or removing a server reshuffles the mapping for everyone.
The better answer is usually to make the servers stateless — push session state to a shared cache — and then use an algorithm that actually balances. Reach for IP hash when you cannot change the application.
Static versus dynamic
Algorithms are categorized by whether they consider the machine's state:
| Static | Dynamic | |
|---|---|---|
| Example | Round robin | Least connections |
| Basis for the decision | Fixed configuration | Current state of the servers |
| Overhead | Low — simple and cheap | Higher — complexity plus communication |
| Accounts for health and load | ||
| Behavior with a slow server | Keeps sending it traffic | Routes around it |
Static algorithms distribute tasks based on fixed configurations. They are simple and low-overhead but do not account for real-time server health or load.
Dynamic algorithms monitor the current state of servers. They add complexity and communication overhead, but they improve reliability by routing around overloaded or unhealthy machines. Dynamic algorithms require load balancers to exchange information, which makes them modular but complex — and gives them better forwarding decisions through active health monitoring.
The distributed-state problem
The failure is not that the algorithm is wrong — it is that every balancer runs it correctly on incomplete information, and they all reach the same conclusion at the same moment.
Dynamic algorithms have a subtlety worth raising, because interviewers probe it.
"Least connections" is exact when one load balancer sees all traffic. With several load balancers in front of the same pool, each knows only its own connections — so all of them may independently conclude the same server is least loaded and send it a simultaneous burst. This is a herd effect created by the balancing algorithm itself.
Going deeper (Staff+): the power of two choices
The standard fix is elegantly cheap. Instead of picking the globally least-loaded server, pick two at random and send the request to the less loaded of the two.
This performs dramatically better than pure random, gets close to true least-connections, and — crucially — avoids the herd, because two load balancers are unlikely to sample the same pair. It also needs no global state, which makes it robust when the state you have is stale.
"I'd use power-of-two-choices rather than strict least-connections, because with multiple LBs the global view is stale and strict least-loaded creates herding" is a genuinely senior answer.
Choosing one
| Workload shape | Algorithm | Why |
|---|---|---|
| Identical servers, uniform short requests | Round robin | Cheapest thing that works |
| Mixed server capacities | Weighted round robin | Weights encode the capacity difference |
| Highly variable request duration | Least connections | In-flight count tracks real occupancy |
| Latency-sensitive, heterogeneous fleet | Least response time | Directly optimizes the metric you care about |
| Sticky sessions with no shared store | IP hash | Deterministic client-to-server mapping |
| Distinct workloads behind one domain | URL hash | Routes each path to its specialized cluster |
| Many LBs, large pool | Power of two choices | Near-optimal without global state or herding |
Key takeaway
Start with round robin, move to a dynamic algorithm the moment request durations vary or servers stop being interchangeable — which, at any real scale, is immediately.
Interview signal by level
| Level | What a strong answer sounds like |
|---|---|
| L4 | Names algorithms: "round robin, or least connections." |
| L5 | Matches to the workload: "request durations vary a lot here, so least connections — round robin would pile work onto a server already running a long job." |
| Staff+ | Handles the distributed case: "with several LBs, strict least-connections herds because each has a stale local view. Power-of-two-choices gets near-optimal balance with no shared state. And I'd rather make servers stateless than use IP hash for stickiness." |
Next: what happens when the load balancer has to remember something.