The problem: work in the wrong place
Two jobs are still done while the user waits, and both hurt.
Counting clicks. Every redirect runs UPDATE links SET clicks = clicks + 1 before answering. At 100,000
redirects a second that's 100,000 writes a second on the primary, which was built for about 350. Worse, a viral link
gets thousands of updates a second to the same row, and each one waits for the one before it to release the row.
Redirects for that link slow to a crawl at exactly the moment it matters most.
Checking new links. Snip checks every new long URL against an outside service that lists malware and phishing sites. It usually answers in 200 ms, sometimes takes two seconds, and now and then is down. While the check runs, the owner waits. When the service is down, nobody can create links.
Neither job's result is needed to answer the request. The redirect only needs the long URL; the owner only needs the short code.
The idea: do it later
Put the work on a queue and answer straight away. A queue is a list of messages that keeps them safely until something takes them off. Separate programs called workers take messages from the queue and do the work at their own pace.
4 steps. Use Next and Back to follow the request.
Text version
- A redirect looks up the long URL in Redis, as before.
- Instead of updating the count, the app adds a small message to the queue (code, time, country) and sends the 302 right away. Adding a message takes about a millisecond.
- Workers take messages off in batches of a few thousand and add them up in memory: x7Kp +1,840, b2Qe +12, and so on.
- Each batch becomes one UPDATE per code. Thousands of clicks on a viral link become a single write, a few seconds late, which the requirements allow.
- Snip app → Redis: look up
- Snip app → Click queue: click event
- Click queue → Stats workers: batches
- Stats workers → Postgres: one UPDATE per code
The link check works the same way, with a twist that's worth saying out loud. The link is saved and works immediately (a requirement), and a check message goes on a second queue. Within seconds a worker checks it; if the URL is dangerous, the worker disables the link and deletes its cache entry. The cost of that choice: a bad link can work for a few seconds. The alternative, making every owner wait for the check and blocking creation when the service is down, was judged worse. A stricter product could choose differently.
What a queue buys
- Fast requests. The user waits for the append, not the work.
- Smoothed spikes. If clicks jump tenfold for an hour, the queue grows and workers catch up afterwards. The database sees a steady stream instead of the spike.
- Isolation from failures. When the link-check service is down, messages wait in the queue and are retried later. Link creation keeps working.
- Separate scaling. Add workers when the queue grows, without touching the app servers.
What it costs
- Results arrive later. Counts are seconds behind, and a link is checked a moment after it starts working. Only move work whose result can wait.
- Duplicates. Most queues guarantee each message is delivered at least once, not exactly once. If a worker adds a batch to the counts and then crashes before telling the queue it's done, the batch is delivered again and counted twice. Workers need to be safe to repeat, for example by recording which batches they've applied. That's 07.03 Delivery guarantees and idempotent consumers.
- Order isn't guaranteed unless you design for it (07.04 Partitions, consumer groups and ordering).
- Something new to watch. Track how many messages are waiting and how old the oldest one is. A growing queue means workers can't keep up. Messages that fail again and again go to a separate dead-letter queue so they don't block the rest.
Which work should be async?
| Snip task | Sync or async | Why |
|---|---|---|
| Redirect | Sync | The user is waiting for the answer. |
| Create a link | Sync | The owner needs the short code now. |
| Count a click | Async | Counts may lag a minute; doing it inline is the bottleneck. |
| Check a URL for malware | Async | Slow, unreliable outside service; a few seconds' delay is acceptable. |
| Email a weekly stats report | Async, on a schedule | Nobody is waiting; it can run at 3 a.m. |
A good rule: if the user can't continue without the result, do it now. If they can, it's a candidate for the queue.
Snip at the end of Module 02
Here's everything this module added, with a redirect and a link creation walked through it:
5 steps. Use Next and Back to follow the request.
Text version
- The page's scripts and styles come from a nearby CDN edge (02.05).
- A redirect reaches one of ten stateless app servers through the load balancer (02.03).
- The app finds the long URL in Redis; on a miss it reads a replica, and the primary only for brand-new codes (02.04).
- It queues a click event and answers with a 302 (02.06).
- Workers fold clicks into counts in batches, and check new links against the outside service, disabling any that are dangerous.
- Browser → CDN: files
- Browser → Load balancer
- Load balancer → Snip app
- Snip app → Redis
- Snip app → Postgres
- Snip app → Queues
- Queues → Workers
- Workers → Postgres
- Workers → Safety check
The one part that hasn't been scaled out is the Postgres primary: every write still goes through one machine. At 350 new links a second it's comfortable. If Snip grew another hundredfold, writes and storage would outgrow one machine, and the data would have to be split. That's Part 2: 04.07 Replication: leaders, followers and lag for replication done properly, 05.01 Sharding: range, hash, directory for splitting data, and 05.05 Distributed ID generation for generating unique short codes without one central counter.
Seen in production
GitHub
2009Resque
GitHub built Resque, a Redis-backed job queue, after its previous database-backed queue made creating a job take two seconds or more, time added to every page that created one. GitHub said it relied "very, very heavily" on its queue: Resque had processed over 10 million jobs since a recent data centre move, with separate queues so repository creation and forks ran ahead of jobs like warming caches, and archive jobs ran only on the file servers.
What we foundMoving work to a fast, Redis-backed queue took seconds off page loads, and priorities kept important jobs ahead of housekeeping.
Introducing Resquegithub.blog ↗Snip's stats workers crash and stay down for ten minutes during peak traffic. What do users notice?
Try it
Take the Pastebin from 01.07 The design framework, step by step (a million new pastes a day, 10 reads each, pastes up to 1 MB) and grow it the way this module grew Snip. Start with one box. Then say what you'd add, in what order, and which number or requirement forces each step. Draw the final design.
Recap
Interview questions
When would you use a message queue in a design? What problems does it introduce?
How would you count clicks on a link that suddenly gets 50,000 clicks a second?