Tail latency and why p99 matters

Why the slowest 1% matters more than it sounds

Snip promised redirects under 50 ms at p99: 99 in 100 requests faster than that. It's tempting to treat the other 1% as noise. In a large system it isn't, for two reasons.

Users make many requests. Opening Snip's dashboard loads the page, its scripts, the link list and stats: maybe 20 requests. If each has a 1% chance of being slow, about 18% of dashboard loads hit at least one slow one. The p99 of a request becomes the typical experience of a page.

One request often waits for many servers. Snip's stats are split across 100 shards. The dashboard's "clicks for all my links" query asks every shard and waits for all of them. Now the request is as slow as the slowest of 100.

The arithmetic is the same in both cases: if each part is slow with probability p, a whole made of N parts is slow with probability 1 − (1 − p)ᴺ. Try it:

With 100 servers each slow 1% of the time, 63% of whole requests are slow. A server-level p99 problem has become the common case.

Where the tail comes from

Most slow requests aren't caused by the request itself. They're caused by what else the server was doing:

• Queueing: a burst arrives and requests wait behind each other (03.11 Backpressure and load shedding).
• Pauses: garbage collection, a background compaction, a log rotation.
• Shared machines: another program on the same host using the CPU, disk or network.
• Cold caches: the first request after a restart or eviction goes to disk.
• Retries: a request that needed a retry took at least two round trips (03.09 Timeouts, retries, backoff and jitter).

Some of these you can reduce (tune GC, schedule background jobs at quiet times, isolate noisy workloads), but you can't remove them all. At scale, something is always having a bad moment.

Cutting the tail

Since you can't make every server fast all the time, design so one slow server doesn't make the request slow:

• Hedged requests. Send the request to one replica. If it hasn't answered by about the 95th-percentile time, send a second copy to another replica and use whichever answers first. Only the slow 5% get a second copy, so extra load is small, but a request is now slow only if both copies are.
• Fewer, bigger fan-outs. Asking 10 servers instead of 100 cuts the odds of meeting a slow one tenfold. Keep data a request needs together where you can.
• Partial results. For some features, "results from 98 of 100 shards, now" beats "all 100, in two seconds". Set a deadline and answer with what you have, saying so if it matters.
• Separate slow work. Keep bulk and background work off the servers that answer users, or behind lower priority.

Measure it properly

• Look at percentiles, not averages. An average hides the tail entirely.
• Don't average percentiles. The p99 of a whole fleet isn't the average of each server's p99. Collect latency histograms and combine those.
• Measure from the client's side too. Server timing misses time spent queued before the server picked the request up, which is often where the tail lives.

Recap

Interview questions