The problem: one agency slows down everyone
Snip's 40,000 paying customers share the same servers, queues and databases. One agency launches a campaign that sends a burst of 2 million link creations through the API, plus an export of three years of click history. For an hour, every other customer's link creation is slow and their exports sit queued. Nobody broke a rule; one tenant simply used a lot.
That's the noisy neighbour problem, and every shared (multi-tenant) system has it.
How much to share
| Model | What tenants share | Isolation | Cost per tenant |
|---|---|---|---|
| Pooled | Everything: servers, tables (with a tenant ID column), queues | Lowest | Lowest |
| Partitioned | Infrastructure, but each tenant's data in its own schema, shard or queue | Medium | Medium |
| Dedicated (silo) | Nothing: a separate stack per tenant | Highest | Highest |
Most products mix them: pooled for the long tail, dedicated for the largest customers or those with strict compliance needs. Snip pools everyone, gives its top 20 agencies their own database shards and search indexes (10.04 Sharding a search cluster), and offers a dedicated deployment on its enterprise plan.
Limits per tenant
The first defence is making sure no tenant can take more than its share:
- Rate limits per tenant on every expensive operation (14.03 Rate limiting algorithms), set by plan.
- Concurrency limits per tenant: at most 5 exports running at once per customer, however many they request.
- Fair queuing: instead of one first-in-first-out queue, round-robin between tenants' queues, so one tenant's 10,000 queued exports don't sit in front of everyone else's one.
- Quotas on size: links per account, storage per account, so growth is a conversation, not an incident.
Cells: cap the blast radius
Limits stop one tenant from using too much; they don't help if a tenant triggers a bug that crashes the servers they run on. Cells split the system into several complete, independent copies, each serving a subset of tenants. A poison request or a bad deploy (14.05 Safe deploys) can only take down one cell: a fraction of customers instead of all of them. Cells also cap how big any one deployment gets, so each is tested at a known scale.
Shuffle sharding
Cells divide tenants into fixed groups, so each tenant shares a fate with everyone in its cell. Shuffle sharding does better. Give each tenant a random combination of a few workers out of many. Two tenants rarely get exactly the same combination, so a tenant that poisons its workers takes out only its own combination; other tenants usually share at most one of those workers, and still have the rest of theirs.
Amazon's numbers: with 8 workers split into 4 regular shards of 2, a bad tenant affects 25% of tenants. With shuffle shards of 2 out of the same 8 workers, there are 28 combinations, and the impact drops to about 1/28th. Route 53 uses 2,048 virtual name servers and gives each domain a shuffle shard of 4, which allows about 730 billion combinations.
Seen in production
Amazon
2019Shuffle sharding
Shuffle sharding creates virtual shards, for example of two workers each, and assigns each customer to one. With eight workers there are 28 unique pairs, so a problem's scope of impact is about 1/28th, seven times better than regular sharding into four shards, where it would be 25 percent. For Route 53, Amazon arranged capacity into 2,048 virtual name servers and gave each domain a shuffle shard of four, for about 730 billion possible combinations.
What we foundRandom combinations of workers per customer shrink a bad customer's blast radius from a shard to a sliver.
Workload isolation using shuffle-sharding (Amazon Builders' Library)builder.aws.com ↗Snip's export workers take jobs from one shared queue in order. An agency queues 10,000 exports. What do other customers experience, and what's the best fix?
Recap
Interview questions
How do you prevent one customer from degrading service for others in a multi-tenant system?