Stateless servers behind a load balancer

The problem: one app server

Snip's app server is still alone. When it crashes, every redirect fails, and every deploy means a few seconds of errors while it restarts. Snip promised redirects would almost never fail, so the fix is the same as for the database: more than one copy, so any one can fail or be restarted while the others answer.

Two copies need something in front of them to decide which one gets each request. That's a load balancer.

The load balancer

A load balancer is the one address the outside world talks to. It receives each request and passes it to one of the app servers behind it:

It does three jobs:

• Spreads requests. The simplest rule is round robin: first request to server 1, next to server 2, and so on. "Fewest open connections" is a common alternative when requests vary in cost. 03.06 Load balancers covers the options.
• Checks health. Every few seconds it asks each server a simple question, usually an HTTP request to a path like /health. A server that fails a few checks in a row stops getting traffic until it recovers. This is what turns "one server crashed" from an outage into a non-event.
• Hides the servers. Users only know the load balancer's address, so you can add, remove and replace servers behind it without anyone noticing.

That last point also gives Snip deploys without downtime: take one server out of rotation, update it, check its health, put it back, and repeat for the next. At every moment at least two servers are answering.

The catch: state

This all works only if any server can handle any request. Here's what goes wrong if one doesn't. Say the app keeps logged-in sessions in its own memory: after you log in, server 1 remembers "this cookie belongs to Ana". Now step through Ana's next two requests:

Anything a server keeps in its own memory or on its own disk between requests is state: sessions, uploaded files waiting to be processed, counters, an in-memory list of rate limits. With state on the servers, they're no longer interchangeable, and losing one loses its state.

There are two ways out.

Sticky sessions. Tell the load balancer to always send the same user to the same server, usually with a cookie of its own. It's quick to set up, but it treats the symptom: load spreads unevenly, and when a server dies, everyone stuck to it still loses their session.

Stateless servers. Move the state out of the app servers into something they all share:

Now each server is stateless: it keeps nothing between requests that another server would need. Any server can answer any request, a crash loses nothing, and adding capacity means starting another copy. This is the version Snip uses. A Snip redirect was stateless from the start (look up the code, answer); only the logged-in dashboard needed fixing.

The load balancer is a single point of failure too

Putting everything behind one load balancer just moves the problem. In practice you either run a pair of them, where a standby takes over the shared address if the active one fails, or use a cloud provider's managed load balancer, which is already run as many machines behind the scenes. Either way, it's worth saying in an interview.

Recap

Interview questions