Backpressure and load shedding

The problem: too much at once

A celebrity posts a Snip link to 50 million followers, and redirects jump to three times Snip's usual peak. Snip can add servers, but that takes minutes, and the spike is now.

Without a plan, here's what happens. Requests arrive faster than servers finish them, so they queue. Each request waits longer, until it's waiting longer than the client's timeout. The client gives up and retries, adding more load. The server eventually reaches each abandoned request and does the full work anyway, for nobody. It's busy at 100%, and almost none of its work is useful.

That's the key distinction:

• Throughput is how many requests a second the server handles.
• Goodput is how many it handles in time for the client to use the answer.

Past capacity, throughput can stay high while goodput collapses towards zero. The goal of this lesson is to keep goodput at capacity even when demand is far beyond it.

Backpressure: bounded queues

Every queue in a system should have a limit. When it's full, the producer is told so, and slows down or fails fast, instead of the queue growing without end. That push upstream is backpressure.

• A server's request queue that's full answers new requests with 503 straight away.
• A job queue that's full makes producers wait, or reject new jobs, instead of holding hours of backlog.
• A client that sees backpressure (a 503 or 429, maybe with Retry-After) backs off (03.09 Timeouts, retries, backoff and jitter).

Unbounded queues feel safe ("we never drop anything!"), but all they do is turn an overload into very high latency and, eventually, running out of memory.

Load shedding: say no early and cheaply

Load shedding is a server deciding, as each request arrives, whether it has room to serve it well. If not, it rejects it immediately with a cheap error, before doing any real work. A rejected request costs microseconds; an accepted one that times out costs the full work and helps nobody.

Common signals for "no room": too many requests in progress, a queue past its limit, CPU above a threshold, or recent latency climbing towards the timeout.

Not all requests are equal

When you must turn some work away, choose which. Snip ranks its traffic:

As load rises, Snip sheds from the bottom of the table up.

Don't work on requests nobody is waiting for

Two more habits keep goodput high:

• Deadlines. The client sends how long it will wait (or each hop passes on the time remaining). A server that picks up a request whose deadline has already passed drops it instead of doing it. A request halfway through that runs out of time stops early.
• Graceful degradation. Instead of failing outright, return something cheaper: a cached stats page from a minute ago, the dashboard without its live counter, the redirect without recording the click.

Recap

Interview questions