The problem: the shards are filling up
Snip runs 256 logical shards on 8 Postgres machines, 32 each. Two years on, the machines are at 80% disk and CPU climbs every month. The plan: 16 machines, 16 logical shards each. That means moving 128 logical shards, half the data, onto new machines, while redirects keep working and no link is lost.
Rebalancing has three requirements:
- Load ends up even across the new set of machines.
- Only as much data moves as necessary. The new machines must receive half the data; moving more is waste.
- The system stays up, and every write made during the move ends up in the right place.
How the data is divided decides how it can move
| Scheme | How you grow | Verdict |
|---|---|---|
| hash mod N machines | Change N: nearly every row moves | Avoid |
| Fixed number of logical partitions | Move whole partitions to new machines; the key → partition mapping never changes | The common choice: Notion, Dynamo's final design, Snip |
| Split ranges when they grow | A range past its size limit splits in two, and one half can move | Natural for range sharding; partitions follow the data's size |
| Consistent hashing | New nodes take a share of the ring from their neighbours | Common in caches and Dynamo-style stores |
With fixed logical partitions, choose the count up front, and generously: it caps how many machines you can ever spread over. Notion picked 480 because it divides evenly into many machine counts. Too many partitions has a cost too, since each one carries overhead (connections, files, metadata), so a few hundred to a few thousand is typical.
Moving one logical shard
Each move follows the same routine, close to the expand and contract pattern of 04.09 Schema changes and online migrations:
- 01
Copy
Snapshot shard 41's tables onto the new machine, throttled to spare the old one.
- 02
Catch up
Stream every change made since the snapshot, via logical replication.
- 03
Verify
Compare row counts, checksums, and sampled reads from both copies.
- 04
Cut over
Pause shard 41's writes, wait for lag to reach zero, flip the map, resume.
- 05
Keep a way back
Replicate new writes back to the old copy, in case you must roll back.
- 06
Clean up
After a quiet period, drop shard 41 from the old machine.
The cutover is the only moment that touches live traffic. Writes for one logical shard pause for a second or two in the router or connection pooler, while the last changes replicate, and then resume against the new machine. Reads can often keep going throughout.
Then repeat, a few shards at a time, watching load on both sides. Snip moves 128 shards over a couple of weeks, in daytime, with a person watching.
Automatic or by hand?
Some databases rebalance automatically when a node joins, leaves or fills up. That's convenient, and also risky: a rebalance is heavy work, and if a slow node is mistaken for a dead one, the system may start moving its data onto nodes that are already busy, making everything slower and triggering more moves. Many teams let the system propose a plan and have a person approve it, and they throttle moves so they can't crowd out real traffic.
Seen in production
Notion
2023Postgres, 32 to 96 machines
Notion's shards were running above 90% CPU at peak, so it tripled its machines from 32 to 96, going from 15 logical shards per database to 5. Data moved with Postgres logical replication (creating indexes after the copy cut syncing from three days to twelve hours). Dark reads compared old and new copies. Each cutover paused traffic in PgBouncer, confirmed replication had caught up, switched the shard mapping, set up replication back to the old databases for rollback, and resumed. At worst, users saw a saving spinner for about a second.
What we foundLogical shards made it a move, not a re-split: logical replication, dark reads, then a per-database cutover users barely noticed.
The Great Re-shard: adding Postgres capacity (again) with zero downtimenotion.com ↗Slack
2020Vitess resharding
In March 2020, as people moved to working from home, Slack's query rates rose by 50% in a single week. One busy keyspace was resharded using Vitess's splitting workflows, which copy, catch up and cut over for you.
What we foundA sharding layer with built-in split workflows turned a sudden surge into routine work.
Scaling Datastores at Slack with Vitessslack.engineering ↗Snip has 256 logical shards on 8 machines and wants 16 machines. With fixed logical shards, how much data moves?
Recap
Interview questions
Your sharded database needs twice as many machines. How do you move the data without downtime?