Timeouts, retries, backoff and jitter

Timeouts: never wait forever

Snip's workers call an outside safety-check service for every new link (02.06 Async workers and queues). One afternoon the service doesn't fail; it just gets slow, taking 60 seconds instead of 200 ms. Workers with no timeout sit waiting, one by one, until every worker is stuck and the check queue backs up for hours.

A slow dependency is often worse than a dead one: a dead one fails fast, while a slow one ties up your threads, connections and memory. So every network call gets a timeout.

How long? Too short and you give up on requests that would have succeeded; too long and you're stuck. A good method: decide what share of good calls you're willing to cut off, say 0.1%, then set the timeout at that percentile of the dependency's latency. If the safety check's p99.9 is 800 ms, a 1-second timeout fails very few healthy calls and caps the damage of a slow day at one second per call.

Retries: often right, sometimes dangerous

Many failures are brief: a dropped packet, a server restarting, a deploy in progress. Trying again a moment later often just works. But a retry is more load on something that may already be struggling, so retry with rules:

1. Only retry what can safely be repeated. Reads, deletes and PUTs are fine. A POST that creates or charges something isn't, unless it carries an idempotency key (03.10 Idempotency keys).
2. Only retry errors that might go away. Timeouts, connection errors, 503 and 429 are worth another try. A 400 will fail the same way every time.
3. Cap the attempts, typically at two or three, and give up with an error after that.
4. Wait between attempts, longer each time, with randomness. That's backoff and jitter, below.
5. Retry at one layer only. See the next section.

The retry storm

Retries multiply across layers. Say each of five layers of services retries three times:

When the database is struggling, it gets 243 times the load from the requests it's already failing, which keeps it down. Two defences:

• Retry at one point in the stack, usually the layer closest to the user or the one that knows whether the operation is safe to repeat. Inner layers fail fast.
• A retry budget. Allow retries only while they're a small share of traffic: a token bucket where each retry costs a token and tokens refill slowly. In a brief blip, every call can retry; in a long outage, retries drop to a trickle automatically.

Backoff and jitter

If a server hiccups and a thousand clients all retry after exactly 100 ms, they arrive together and knock it over again, then again at 200 ms, and so on. Two fixes, used together:

• Exponential backoff: wait longer after each failure (100 ms, 200 ms, 400 ms…), up to a cap, giving the server room to recover.
• Jitter: pick the wait at random within that range, so clients spread out instead of arriving in waves.

async function callWithRetry<T>(call: (signal: AbortSignal) => Promise<T>, { attempts = 3, base = 100, cap = 2000, timeout = 1000 } = {}) {
  for (let attempt = 0; ; attempt++) {
    try {
      return await call(AbortSignal.timeout(timeout)); // every attempt has its own timeout
    } catch (err) {
      if (attempt + 1 >= attempts || !isRetryable(err)) throw err;
      // "Full jitter": wait a random time between 0 and the exponential backoff.
      const backoff = Math.min(cap, base * 2 ** attempt);
      await sleep(Math.random() * backoff);
    }
  }
}

// Timeouts, connection errors, 429 and 5xx might go away; 4xx won't.
const isRetryable = (err: unknown) => err instanceof TimeoutError || err instanceof NetworkError || (err instanceof HttpError && (err.status === 429 || err.status >= 500));

If the server sends a Retry-After header with a 429 or 503, wait at least that long instead.

Circuit breakers, briefly

A circuit breaker watches calls to a dependency, and after too many failures it "opens": calls fail immediately without trying, for a while, then a few test calls are let through to see if it has recovered. It stops a service wasting time and threads on a dependency that's down. The cost is a new mode of behaviour that's hard to test, and which can make a partial outage look total. Many teams prefer retry budgets and timeouts first, and add breakers where they've measured a need.

Recap

Interview questions