Leader and followers
The most common setup has one leader (primary) that accepts every write, and one or more followers (replicas) that receive the leader's log and apply it in order. Followers serve reads, stand by to take over, or both. Snip runs this from 02.02 Split the database; scale up or out onwards: a primary, a standby and a read replica.
The big decision is when the leader says "committed":
| Asynchronous | Synchronous | Semi-synchronous | |
|---|---|---|---|
| Leader waits for | Only its own disk | Every follower to confirm | At least one follower |
| Write latency | Lowest | Highest: slowest follower sets the pace | One extra round trip |
| If the leader dies | Recent writes may exist only on the dead leader | No committed write is lost | No committed write is lost, if that follower takes over |
| If a follower dies | Nothing changes | Writes stop until it's removed | Writes continue if another follower confirms |
Most production setups are asynchronous, or semi-synchronous with one nearby standby. Snip uses a synchronous standby in the same region (so a promoted standby has every committed link: "never lose a link") and an asynchronous read replica (whose lag only affects dashboards).
Lag, revisited
An asynchronous follower is always a little behind. Usually it's milliseconds; under heavy writes, long queries (04.05 MVCC) or network trouble it can be seconds or minutes. 02.04 Read replicas and a cache showed the classic bug, a user not seeing their own write, and the fixes: read from the leader right after writing, or fall back to the leader when data is missing. Another: a user refreshing twice may hit two replicas with different lag and see data go backwards. Pinning each user's reads to one replica avoids that.
Failover
When the leader dies, a follower must be promoted. That takes three decisions, each of which can go wrong:
4 steps. Use Next and Back to follow the request.
Text version
- Detect: the failover manager stops hearing from the primary. But is it dead, or just slow, or cut off from the manager only? It waits for several missed checks to avoid flapping.
- Choose: it promotes the follower with the most of the log. Here the synchronous standby has every committed write, so nothing is lost.
- Redirect: the app is told (via DNS, a proxy or a config change) to send writes to the new primary.
- Re-follow: the read replica starts following the new primary. If the old primary comes back, it must rejoin as a follower, never as a second primary.
- Failover manager → Old primary: health
- Failover manager → Standby: promote
- Snip app → Standby: writes
- Standby → Read replica: log
What can go wrong:
- Lost writes. With asynchronous replication, the promoted follower may be missing the last writes the old leader acknowledged. They're gone, or worse, they come back later and conflict.
- Split brain. If the old leader was only cut off, not dead, it may keep accepting writes while the new one does too. Two leaders, two diverging histories. Systems prevent this with fencing: making sure the old leader can't accept writes (cutting its access, or rejecting writes tagged with an old "term" number).
- Too eager, or too slow. A short timeout fails over on a brief blip, causing more disruption than it prevents. A long one leaves writes down for minutes.
Beyond one leader
Two other designs exist, with harder trade-offs, covered in Part 2's distributed systems module:
- Multi-leader: several nodes accept writes (often one per region) and exchange changes. Writes are local and fast; concurrent writes to the same data conflict and must be resolved.
- Leaderless: clients write to several replicas and read from several, using quorums (08.04 Replication and quorums).
Seen in production
GitHub
2018MySQL with Orchestrator
A network link between GitHub's US East Coast hub and its primary data centre was lost for 43 seconds. Orchestrator began failing database clusters over to the US West Coast, and applications started writing to the new primaries there. The East Coast servers held a brief period of writes that hadn't been replicated, so GitHub couldn't safely fail back, and East Coast applications suffered from cross-country latency. Choosing data integrity over speed of recovery, GitHub ran degraded for 24 hours and 11 minutes, then changed Orchestrator to stop promoting primaries across regions.
What we foundA 43-second network partition triggered a cross-country failover; un-replicated writes on the old primaries made failing back unsafe, and recovery took a day.
October 21 post-incident analysisgithub.blog ↗A primary with one asynchronous replica acknowledges a write, then crashes before sending it. The replica is promoted. What happens to that write?
Recap
Interview questions
Compare synchronous and asynchronous replication. What can go wrong during failover?