sehaj/learn
System design/15 Classic designs

15.01 · design · 20 min

Design: a URL shortener on one machine

The classic first question, answered with the framework: requirements, estimates that say one database is enough, an API, random short codes with tested collision handling, a fast redirect path, clicks counted off the request path, and a standby so one machine failing isn't an outage.

The prompt

"Design a URL shortener." It's the most common opening question because it's small enough to finish in an interview and still has real decisions in it: how codes are made, how redirects stay fast, where clicks are counted, and what happens when the machine fails. This version stays at Snip's launch scale; 15.02 Design: a URL shortener at scale takes it to a hundred times that.

Design it yourself firstjuniormidread-heavyID generationcaching

Users submit a long URL and get back a short one, like snip.example/x7Kp9aQ. Anyone who opens the short link is sent to the long URL. Assume 10 million new links a month and 100 redirects for every link created. Owners can see how many times each link was clicked.

20:00

1. Requirements

Functional

  • Create a short link for a long URL, optionally with a custom alias (snip.example/spring-sale) and an expiry date.
  • Redirect anyone who opens the short link.
  • Show the owner a click count per link.

Non-functional

  • Redirects under 50 ms at the server for 99% of requests, available 99.99% of the time.
  • A link, once created, is never lost.
  • Codes can't be guessed or enumerated: some links point at private documents.
  • Click counts can lag by a few seconds.

Out of scope: user accounts beyond an API key, detailed analytics, editing a link's destination.

2. Estimates

From 01.05 Back-of-the-envelope estimation: 10 million links a month is about 4 writes a second on average and 10 at peak; with 100 redirects per link, about 350 reads a second and 1,000 at peak; and 600 million links of 500 bytes each over five years is 300 GB.

What it decides: one ordinary Postgres server handles this with room to spare. The design doesn't need sharding, a queue or a cache cluster. It does need a second copy of the data, because 99.99% is about 4 minutes of downtime a month (14.02 SLOs and error budgets), and one machine can't promise that on its own.

3. Entities and API

text
Link   { code (primary key), long_url, owner_id, created_at, expires_at? }
Clicks { code, day, count }

POST /links        { long_url, alias?, expires_at? }   Idempotency-Key: …   →  201 { code, short_url }
GET  /{code}                                                                 →  302 Location: long_url
GET  /links/{code}/stats                                                     →  200 { clicks, by_day }

The create call takes an idempotency key (03.10 Idempotency keys) so a retried request doesn't make a second link. The redirect answers 302, not 301: a 301 is cached by browsers, so later clicks never reach Snip and are never counted, and the destination can never change (01.06 Trade-offs: there is no best design).

4. High-level design

Snip on one machine, plus a standby
One servercheckreplicatearchiveBrowserSnip app+ in-process cachePostgreslinks, clicksStandbystreaming replicaBackupsWAL + nightlyBlocklistBloom filter
Every box traces to a requirement: the standby to availability, the archive to never losing a link, the in-process counters to keeping redirects fast.

5 steps. Use Next and Back to follow the request.

Text version
  1. Create: the app checks the long URL against the phishing blocklist, held in memory as a Bloom filter (05.08).
  2. It draws a random 7-character code and inserts the row. The code is the primary key, so a collision fails the insert and the app draws again.
  3. Redirect: the app looks the code up, first in its in-process cache, then by primary key in Postgres, and answers 302. It adds one to an in-memory counter for the code.
  4. Every few seconds the app writes the counters to the clicks table in one statement, so a thousand redirects cost one write.
  5. Postgres streams every change to a standby on another machine and archives its write-ahead log to object storage, so a lost server costs neither the data nor more than a minute or two of redirects.
  • Inside One server: Snip app, Postgres
  • Browser → Snip app
  • Snip app → Blocklist: check
  • Snip app → Postgres
  • Postgres → Standby: replicate
  • Postgres → Backups: archive

5. Deep dives

Deep dive

How are short codes made?

BadA counter, written in base62

Short and never collides, but guessable: anyone can walk x7Kp, x7Kq, x7Kr and find every link, including private ones, and the codes reveal how many links Snip has.

GoodA hash of the long URL, cut to 7 characters

Not guessable, and the same URL gets the same code. But truncated hashes still collide and need a retry scheme, and two users shortening the same URL now share one link, with one expiry and one click count.

GreatRandom base62 codes, with the primary key catching collisions

62⁷ is about 3.5 trillion codes. With 600 million links, a new random code collides about 0.02% of the time, and the database already checks uniqueness for free: insert, and on a conflict, draw again (05.05 Distributed ID generation). Custom aliases go in the same column, so they're checked the same way.

ts
import { randomInt } from "node:crypto";

const ALPHABET = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz";

export function randomCode(length = 7) {
  let code = "";
  for (let i = 0; i < length; i++) code += ALPHABET[randomInt(62)]; // randomInt avoids modulo bias
  return code;
}

export async function createLink(db: pg.Pool, longUrl: string, ownerId: number, draw = randomCode) {
  for (let attempt = 0; attempt < 5; attempt++) {
    const { rows } = await db.query(
      `INSERT INTO links (code, long_url, owner_id) VALUES ($1, $2, $3)
       ON CONFLICT (code) DO NOTHING RETURNING code`,
      [draw(), longUrl, ownerId],
    );
    if (rows.length) return rows[0].code as string; // no row back: the code was taken, draw again
  }
  throw new Error("no free code after 5 attempts");
}

Tested on Postgres 16: 10,000 links were created in about 0.7 seconds on one connection. With the first draw forced to an existing code, the insert returned no row and the second draw succeeded; with every draw forced to a taken code, the function gave up after five attempts instead of looping forever. The redirect's lookup by code used the primary key index.

Deep dive

Keeping the redirect fast

BadLook up the long URL by scanning, or by an unindexed column

Fine with a thousand links, and seconds per redirect with hundreds of millions.

GoodA primary key lookup on the code

One index lookup, typically well under a millisecond on a warm database, and enough for 1,000 reads a second.

GreatThe primary key, plus a small in-process cache for hot codes

Clicks are skewed: a few links get most of them. An LRU cache of the 100,000 hottest codes (06.03 Eviction: LRU, LFU and TTL) answers most redirects from memory with no database round trip. Give entries a short TTL, such as a minute, so a blocked or expired link stops working quickly.

Deep dive

Counting clicks without slowing redirects

BadUPDATE links SET clicks = clicks + 1 on every redirect

Turns every read into a write, and a popular link's row becomes a lock that every redirect for it waits on.

GoodInsert a row per click

Appends don't fight over one row, and you keep each click's details. But at 350 a second that's 30 million rows a day, which will soon dwarf the links table.

GreatCount in memory, write totals every few seconds

Each app process keeps a map of code → count and flushes it with one upsert every few seconds. The requirements allow counts to lag, and a crash loses a few seconds of counts, never a link. If detailed analytics arrive later, the same flush can write events to a queue instead (02.06 Async workers and queues).

Expiry and aliases. An expired link is checked on read (expires_at < now() answers 404) and deleted later by a nightly job, so expiry never needs a timer per link. Aliases need a reserved list (api, login, admin) and the same character rules as codes.

6. Wrap-up: what breaks first

What's expected at each level

Junior

gets a working design

  • A working API and a table keyed by code
  • Random or hashed codes, with collisions handled
  • Picks a redirect status code

Mid-level

explains the trade-offs

  • Uses the estimate to justify one database
  • Explains 301 against 302 in terms of caching and analytics
  • Keeps click counting off the redirect path; adds idempotent creates

Senior

drives it and goes deep

  • Turns 99.99% into a standby, failover time and backups with a recovery point
  • Covers abuse (blocklists), expiry and aliases
  • Says exactly what changes at 100× and why

Seen in production

Stack Overflow

2016

The whole site

On one day in February 2016, Stack Overflow's load balancers handled over 209 million HTTP requests. The sites ran on 4 Microsoft SQL Servers, 11 IIS web servers, 2 Redis servers, 3 tag-engine servers, 3 Elasticsearch servers and 4 HAProxy load balancers, with Redis instances below 2% CPU.

What we foundA very large site ran on a small, carefully tuned set of servers; they scaled up and tuned before they scaled out.

Stack Overflow: The Architecture – 2016 Editionnickcraver.com ↗
Predict first

An interviewer asks why Snip doesn't use an auto-increment ID in base62 as the code. It's shorter and never collides. What's the strongest answer?

Commit to an answer before you read on.

Recap

Interview questions

JuniorfreqMust-knowDesign

Design a URL shortener.

Report a mistakeEdit this page