Load balancers

Two kinds of load balancer

A load balancer can make its decision at two different levels, named after layers of the network model:

• Layer 4 (transport) looks only at connections: IP addresses and ports. It picks a server when a TCP connection opens and passes the bytes through without reading them. Very fast and protocol-agnostic, but it can't see URLs, headers or cookies, and every request on one connection goes to the same server.
• Layer 7 (application) ends the connection itself, reads each HTTP request, and decides per request. It can route /api/* to one group of servers and /static/* to another, add headers, terminate TLS, retry a failed request on another server, and spread requests from one long-lived connection across many servers.

Large sites often use both in layers: a layer 4 tier spreads connections across a fleet of layer 7 proxies, which then route requests to applications. GitHub's setup below is exactly this.

Choosing a server

The algorithm matters most when requests or servers aren't all the same. The common ones:

• Round robin: each server in turn. Even when every request costs the same, and blind to everything else.
• Weighted round robin: bigger servers get proportionally more turns.
• Random: surprisingly decent, but uneven over short stretches.
• Least connections: the server with the fewest requests in progress right now. Adapts to slow requests and slow servers, because a server that's struggling holds more open requests and so gets fewer new ones.
• Two random choices: pick two servers at random and send to the less busy one. Nearly as good as least connections, and it doesn't need every load balancer to have a perfectly up-to-date view of every server, which matters when there are many load balancers.
• Hashing (by client IP, user ID, or URL): the same key always goes to the same server. Useful when servers keep a per-key cache. Plain hashing reshuffles almost every key when a server is added; consistent hashing fixes that (05.03 Consistent hashing).

Try them. Four servers, requests arriving at random, each server working through its own queue one request at a time:

The slow-server case is the one to remember. Round robin keeps sending a quarter of all traffic to a server that can only handle a seventh of it, so its queue grows without limit. Algorithms that look at how busy each server is route around the problem automatically.

Health checks, in more detail

Health checks decide which servers are in the pool at all. Two kinds work together:

• Active checks: the load balancer requests /health every few seconds, and removes a server after, say, three failures in a row; it adds it back after a few successes. The thresholds stop one slow check from flapping a server in and out.
• Passive checks: the load balancer watches real traffic, and backs off a server that's returning errors or timing out, without waiting for the next check.

Be careful what /health tests. If it checks the database and the database blips, every server fails its health check at once and the load balancer has nothing left to send traffic to. Many teams keep the health check to "this process can serve requests" and handle dependency failures in the request path instead (03.11 Backpressure and load shedding).

Draining: removing a server politely

To deploy or retire a server, you drain it: stop sending it new requests or connections, let the ones in progress finish (up to a time limit), then take it out. Without draining, every deploy cuts off requests mid-flight. For long-lived connections like WebSockets (03.05 Realtime: polling, SSE and WebSockets), draining can take a long time, so servers usually close connections gently and let clients reconnect elsewhere.

Recap

Interview questions