The problem: Virginia goes dark
Snip runs in two regions (08.02 Consistency models), but until now, the second one only served reads; all writes went to Virginia. What happens if Virginia, all of it, becomes unreachable for six hours? And, more commonly, what happens when one of its three availability zones fails?
Disasters come in sizes, and each size has its own answer.
Losing a zone: static stability
A region has several availability zones: separate data centres with independent power and networking, close enough for fast replication. Spreading servers and replicas across three zones means losing one is routine.
The subtle part is what recovery requires. If surviving zones must launch new servers to absorb the lost zone's load, recovery depends on the control plane that launches servers working perfectly during an incident, exactly when it's busiest. A statically stable design needs no such action: each zone runs with enough spare capacity that the other two can carry the full load as they are. With three zones, that means running about 50% over, each zone at about two-thirds of what it can take. Amazon describes this as the system continuing to work even when a dependency becomes impaired.
Losing a region: four strategies
From cheapest and slowest to most expensive and fastest:
| Strategy | Second region holds | Recovery time | Data loss |
|---|---|---|---|
| Backup and restore | Backups only | Hours to days: rebuild everything | Since the last backup |
| Pilot light | Replicated data, servers switched off | Tens of minutes to start servers | Replication lag |
| Warm standby | A scaled-down, running copy | Minutes: scale it up | Replication lag |
| Active-active | A full copy, serving traffic | Near zero | Near zero for local writes |
Snip chooses warm standby for its write path: Frankfurt runs a small copy of every service and a replica of every database. Failing over means promoting Frankfurt's replicas to primaries, scaling up, and pointing traffic there. Redirects are already active-active, served from both regions' caches and replicas.
Data loss is replication lag
Cross-region replication is asynchronous: waiting for an ocean round trip on every write would slow everything (08.03 CAP and PACELC, without the myths). So whatever Virginia committed but hadn't yet shipped to Frankfurt is lost if Virginia vanishes. Snip's cross-region lag is usually under a second, so its RPO for a region loss is about a second, a few minutes in a bad case. For writes that can't be lost, the options are synchronous cross-region commits (slow) or designs where each region owns some data and accepts some unavailability during a split.
Failover is a procedure, not a switch
- Decide who decides. Automatic regional failover risks flapping on a false alarm (08.01 Partial failure); most teams make it a deliberate human decision with a pre-written runbook.
- Avoid split-brain. Before promoting Frankfurt, make sure Virginia can't also take writes when it comes back (fencing, 08.08 Distributed locks and fencing tokens).
- Fail back carefully. Writes made in Frankfurt during the outage must be reconciled with whatever Virginia had that never replicated.
- Practise. Fail over in a planned exercise every quarter. A failover that's never been run will fail in some way nobody predicted.
Seen in production
Amazon
2019Static stability
In a statically stable design, the overall system keeps working even when a dependency becomes impaired. Rather than launching new instances when a zone fails, a statically stable service overprovisions so it continues operating correctly without launching any. With three availability zones, that means overprovisioning by 50 percent, each zone running at about 66 percent of the level it has been load-tested for.
What we foundProvision enough in each zone that losing one needs no new capacity: with three zones, about 50% over.
Static stability using Availability Zones (Mike Furr, Amazon Builders' Library)builder.aws.com ↗Amazon
DocsDisaster recovery strategies
The whitepaper describes four approaches. With backup and restore, backup frequency determines the achievable recovery point. Pilot light keeps data replicated and core infrastructure available with servers switched off, so it can't serve requests without action first; warm standby runs a scaled-down copy that can handle traffic immediately at reduced capacity. Multi-site active/active can reduce recovery time to near zero for most disasters.
What we foundBackup and restore, pilot light, warm standby, and multi-site active/active, in order of cost and recovery speed.
Disaster recovery options in the cloud (AWS whitepaper)docs.aws.amazon.com ↗Snip's three-zone service runs each zone at 90% of its capacity, relying on autoscaling to replace a lost zone. A zone fails during a large outage of the cloud provider's control plane. What happens?
Recap
Interview questions
How would you design a service to survive the loss of an entire region?