Safe deploys

The problem: the deploy at 13:58

Snip's lunchtime incident (14.01 Observability: logs, metrics, traces) began with a deploy: a migration dropped an index, and redirect latency climbed. The deploy went to all 40 servers in two minutes, and it took the team half an hour to notice and roll back. Most incidents start like this, with a change. The goal isn't to stop changing; it's to make every change small, observed and quickly reversible.

Ways to replace running code

• Rolling: replace servers a batch at a time, behind the load balancer. Old and new versions run side by side for a while, so every change must work with both (04.09 Schema changes and online migrations).
• Blue-green: bring up a full new fleet (green) beside the old (blue), then switch traffic over in one step. Rolling back is switching back. It needs twice the capacity during the switch.
• Canary: send a small share of traffic (1%, then 5%, then 25%) to the new version, compare its errors and latency with the old version's, and continue only if they match.

Waves, bake time and automatic rollback

Amazon's deployment pipelines, described by Clare Liguori, show the full shape:

1. One box first. Deploy to a single server, or the smallest unit, serving at most about 10% of a region's requests, and watch its error rate specifically.
2. Waves of increasing size. Then a first region, then groups of regions, never everything at once.
3. Bake time. Wait between waves, so slow-burning problems (memory leaks, daily jobs) have time to show: at least an hour after the one-box stage and at least 12 hours after the first regional wave.
4. Automatic rollback. The pipeline watches alarms, and if they fire, it rolls back on its own, without waiting for a person.

Snip's smaller version: one server, wait 15 minutes, 25% of servers, wait 30 minutes, the rest; the pipeline rolls back if the canary's error rate or p99 is worse than the rest of the fleet's.

Feature flags: deploy is not release

A feature flag wraps new behaviour in a runtime switch. The code ships turned off, then is turned on separately: for staff, then 1% of users, then everyone. Turning a misbehaving feature off takes seconds, with no deploy. It also makes big changes into many small ones, since unfinished work can be merged and deployed dark.

Flags have costs: two code paths to test, and stale flags that linger for years. Give each a planned removal date.

Global config is a deploy too

Configuration, feature flags, firewall rules and data files can break everything as fast as code, and are often pushed everywhere at once. Cloudflare's July 2019 outage was caused by a single firewall rule with a regular expression that backtracked catastrophically, pushed globally in one step; CPU on every core of every server hit 100% and traffic was down for 27 minutes. Among its fixes: roll out rule changes in stages, like other software.

Recap

Interview questions