Partial failure

The problem: did it happen or not?

Snip's admin tool blocks a phishing link by calling the links service: POST /links/x7Kp/block. The call times out after two seconds. The admin wants to know one thing: is the link blocked?

There's no way to tell from the timeout. The request may never have arrived. It may have arrived and the service crashed before acting. It may have blocked the link and the reply was lost. Or the service is just slow, and it will block the link a second from now. A program on one machine never has this problem: a function call returns or throws. A call across a network has a third outcome, unknown.

Eight places for one call to go wrong

Amazon's Jacob Gabrielson lists the steps every request and reply goes through, at a minimum: post the request, deliver it, validate it, update the server's state, post the reply, deliver it, validate it, update the client's state. Each can fail on its own, and for most of them the client sees the same thing: no answer. As the article puts it, the result of any network operation can be unknown, "in which case the request may have succeeded, failed, or received but not processed".

Partial failure

In a distributed system, some parts fail while others keep working. That's partial failure, and it comes in more forms than "crashed":

• Crash: the process stops. The easy case, if you can tell.
• Omission: messages are dropped, in one direction or both.
• Slowness: the node works, just too slowly to be useful. Often called a gray failure, because health checks pass while users suffer.
• Partition: the network splits, and each side thinks the other died.
• Wrong clocks: the node works, but its idea of the time is off (05.06 Clocks, skew and ordering IDs).

The hard truth is that a slow node and a dead node look the same from outside. A timeout is a guess: "it's been long enough that I'll act as if it's gone". Guess too soon and you fail over from a healthy node; too late and users wait.

Habits that cope

1. Assume unknown, and make retries safe. Every call that changes something gets an idempotency key or is naturally repeatable (03.10 Idempotency keys, 07.03 Delivery guarantees and idempotent consumers). Then "unknown" has a simple answer: try again.
2. Ask, don't assume. After a timeout, the admin tool can read the link's status before retrying, or show "blocking…" until a later check confirms.
3. Time out everything, with limits based on measured latency, not hope (03.09 Timeouts, retries, backoff and jitter).
4. Expect slow, not just dead. Watch latency percentiles, not just up/down health checks, and stop sending work to a node that's slow (08.09 Circuit breakers and bulkheads).
5. Don't let one failure spread. Bound queues, isolate resources and shed load (03.11 Backpressure and load shedding), so a sick dependency doesn't take everything with it.
6. Test the failures. Kill processes, drop packets, add delay and pause nodes in testing. A system tested only on a healthy network has never been tested for the conditions it will meet.

The rest of this module builds on this: how to agree on anything (08.06 Consensus and Raft, intuitively), who's in charge (08.07 Leader election and leases), and what "consistent" can even mean (08.02 Consistency models) when any message might be lost or late.

Recap

Interview questions