The prompt
The follow-up interviewers ask once the one-machine answer is on the board: "Now make it a hundred times bigger, and global." The point isn't a new design, it's knowing which part breaks first and changing only that.
Snip now creates 1 billion links a month, with 100 redirects per link, and half of all clicks come from Asia and Australia. Owners want per-link charts of clicks by minute, country and referrer, at most a few minutes behind. Redirects stay under 50 ms at the server for 99% of requests and available 99.99% of the time.
1. Requirements
The functional list is the same as 15.01 Design: a URL shortener on one machine, plus per-link analytics by minute, country and referrer. The non-functional list changes:
- Redirects under 50 ms at the server for 99% of requests, from every continent, available 99.99%.
- Links never lost; creating a link can take a few hundred milliseconds.
- Analytics at most a few minutes behind; exact totals per day, approximate unique visitors are fine.
- An edited or blocked link stops redirecting within about a minute everywhere.
2. Estimates
| Value | What it decides | |
|---|---|---|
| Writes | 1 billion a month ≈ 400 a second, about 1,200 at peak | Fine for a few databases; the problem is storage |
| Redirects | 100 × writes ≈ 40,000 a second, about 120,000 at peak | Caches must answer almost all of them |
| Storage | 1 billion × 500 bytes = 0.5 TB a month, 30 TB over five years | No longer one database: shard |
| Clicks | 100,000 a second ≈ 8.6 billion a day | A stream with pre-aggregation, not a row per click |
| Codes | 60 billion links over five years | 7 characters collide 1.7% of the time; use 8 (05.05 Distributed ID generation) |
3. Entities and API
The API doesn't change: the clients from 15.01 Design: a URL shortener on one machine keep working. The data gains a second home for the dashboard and the click pipeline:
links { code, long_url, owner_id, created_at, expires_at? } sharded by hash(code)
links_by_owner { owner_id, created_at, code } sharded by owner_id
clicks (stream) { code, at, country, referrer, ua } partitioned by code
clicks_minute { code, minute, clicks, by_country, by_referrer, visitors_hll }4. High-level design
6 steps. Use Next and Back to follow the request.
Text version
- A redirect reaches the nearest CDN edge. Popular links are cached there for 60 seconds, so most redirects end here, near the user.
- On an edge miss, the request goes to the nearest Snip region. The app checks its small in-process cache, then the Redis tier.
- On a cache miss, the app hashes the code to one of 256 logical shards, finds its database in a small routing table, and reads the row by primary key.
- Clicks the origin answers are pre-aggregated per second in each app and published to a clicks topic partitioned by code.
- Clicks answered at the edge arrive later, from the CDN's request logs, and join the same topic.
- Aggregators turn the stream into per-minute rollups, with HyperLogLog sketches for unique visitors, and archive the raw events to object storage.
- Browser → CDN
- CDN → Load balancer: miss
- Load balancer → Snip app
- Snip app → Redis tier: 1st
- Snip app → Link shards: on miss
- Snip app → Clicks topic: clicks
- CDN → Edge logs
- Edge logs → Clicks topic
- Clicks topic → Aggregator
- Aggregator → Analytics DB
5. Deep dives
Deep dive
Storing 30 TB of links
Vertical scaling buys time (02.02 Split the database; scale up or out), but a 30 TB primary means restores that take many hours and one machine whose failure is everyone's outage.
Every redirect touches exactly one shard. But growing from 8 databases to 16 means rehashing and moving most links (05.03 Consistent hashing).
The code is hashed to one of 256 logical shards forever; a small table maps logical shards to physical databases.
Growing means moving whole logical shards with replication and a short cutover (05.04 Rebalancing without downtime), and nothing
rehashes. The dashboard's "my links" reads a links_by_owner copy sharded by owner (05.01 Sharding: range, hash, directory).
import { createHash } from "node:crypto";
export const LOGICAL_SHARDS = 256;
/** Which logical shard owns a code. Never change this: it would move every link. */
export function logicalShard(code: string) {
return createHash("md5").update(code).digest().readUInt32BE(0) % LOGICAL_SHARDS;
}
/** Logical shards map to physical databases through a small table you can change. */
export const physicalFor = (code: string, map: string[]) => map[logicalShard(code)];Tested with a million random codes: every logical shard received between 3,690 and 4,066 of them (the mean is about 3,906). With 8 databases holding 32 logical shards each, moving one logical shard to a ninth database changed where its codes were read from, and nothing else: each code's logical shard stayed the same.
Deep dive
Answering 120,000 redirects a second
120,000 reads a second spread over shards is possible, but every one pays a cross-ocean round trip for half the users, and every cache miss storm lands on Postgres.
Cache-aside (06.02 Cache-aside, write-through, write-behind) with 12 primaries and replicas (06.06 Designing a distributed cache) answers most reads in about a millisecond. But users in Sydney still cross the Pacific to reach it.
Redirects for popular links are cached at the edge for 60 seconds with s-maxage, and kept out of browsers with a
short max-age, so Snip can still purge them. Edits and blocks purge by a surrogate key per link
(06.07 CDNs in depth); the 60-second lifetime caps staleness if a purge is missed.
Deep dive
One link, forty million followers
A celebrity posts a link; 50,000 redirects a second hit one code, one Redis slot and, on a miss, one shard (05.02 Hot partitions and celebrity keys).
The edge absorbs most of it. But each edge location and each cache expiry still sends a burst of misses.
The CDN collapses concurrent misses for the same key into one origin request; app servers keep the very hottest codes in process; and the Redis client coalesces identical in-flight reads (06.05 Stampedes and request coalescing). Clicks for the code are pre-aggregated before they reach the topic, so one hot link doesn't make one hot partition.
Deep dive
Counting 8.6 billion clicks a day
100,000 writes a second, concentrated on popular rows: the click path would become the busiest part of the database.
Off the redirect path, replayable and partitioned by code (07.04 Partitions, consumer groups and ordering). But each click is a message, and a viral link's partition falls behind.
Apps sum clicks per code per second before publishing; consumers write per-minute rollups idempotently (07.03 Delivery guarantees and idempotent consumers), with HyperLogLog sketches for unique visitors (11.03 HyperLogLog); raw events go to object storage for 13 months, so a counting bug can be fixed by replaying them (11.06 Raw events vs derived data).
A second region. Redirects are served active-active from Virginia and Frankfurt, each with its own cache and read replicas. Link creation stays in Virginia, with Frankfurt as a warm standby (14.08 Multi-region and disaster recovery): if Virginia fails, about a second of new links may be lost, and redirects keep working throughout.
Abuse. At this size Snip is a target for phishing. Destinations are checked against blocklists at creation and re-checked in the background, new accounts get low rate limits, and a block purges the link from every cache (14.10 Authentication, authorisation and abuse).
6. Wrap-up
Text version
- Inside Region 1: Virginia, Shards
- Inside Region 2: Frankfurt, Shards
- Browser → CDN
- CDN → Virginia
- CDN → Frankfurt
- Virginia → Shards
- Frankfurt → Shards
- Shards → Shards: async
- Virginia → Click pipeline
- Frankfurt → Click pipeline
What breaks next: the analytics database as owners ask for longer ranges and more dimensions (11.06 Raw events vs derived data), and the cost of the edge logs. And the first thing to say if asked: everything above is an answer to an estimate. At 15.01's scale, none of it would be justified.
What's expected at each level
Junior
gets a working design
- Adds caching and read replicas
- Knows that the database must be split at this size
Mid-level
explains the trade-offs
- Shards by code and handles the owner's dashboard with a second copy
- Layers caches and explains how edits and blocks reach them
- Moves click counting to a stream with batching
Senior
drives it and goes deep
- Uses logical shards so growth never rehashes
- Handles a hot link with edge caching, collapsing and pre-aggregation
- Chooses what's active-active and what's a standby, and states the data loss
- Raises abuse and the cost of the pipeline unprompted
Seen in production
Notion
2021Sharding Postgres
Notion sharded its single Postgres by workspace ID into 480 logical shards, 15 per machine on 32 machines, choosing 480 because it divides evenly into many machine counts. The move used double writes, a three-day backfill and dark reads comparing both databases, then a few minutes of scheduled downtime.
What we foundMany logical shards on fewer machines, with a count chosen to divide evenly as the fleet grows.
Herding elephants: lessons learned from sharding Postgres at Notionnotion.com ↗Discord
2023Message storage
Heavy reads on one busy channel created hot partitions whose latency spread across Discord's database cluster. Discord added data services that coalesce identical requests, querying once for many readers, and routed requests by channel ID with consistent hashing so all requests for a channel reach the same instance.
What we foundOne busy key could slow the whole cluster; routing a key's requests to one service and coalescing them was part of the fix.
How Discord stores trillions of messagesdiscord.com ↗Snip shards links into 16 databases by hash(code) % 16. It needs 24. What happens, and what should it have done?
Recap
Interview questions
How would you scale a URL shortener to 100,000 redirects a second worldwide?