Why move work off the request path

The problem: creating a link got slow

When Snip launched, creating a link was one INSERT. Then the paid plan added features, and each one was bolted on to the same request. Today, POST /links does all of this before it answers:

The p99 for creating a link is now over 5 seconds, and any one of five other systems being down makes it fail. The owner wanted a short code. Everything else is something that should happen because a link was created, not something the owner needs before they can carry on.

Deciding what stays in the request

Ask of each step: does the user need its result to continue?

• Yes, and it must be correct now → do it in the request. The insert stays: without it there's no link.
• No, but it must eventually happen → move it to the background, reliably: the preview, the QR code, the index.
• No, and it's someone else's system → definitely background: the webhook. Never let a customer's slow server slow down your API.
• It decides whether the request is allowed → think hard. The safety check could block the request, but Snip decided in 02.06 Async workers and queues that a few seconds of exposure beats making every owner wait.

After the move, POST /links does the insert and records that background work is needed, and answers in about 10 ms. The work happens a moment later, and the dashboard shows "preview loading" until it's done.

Jobs and events

There are two ways to describe the background work, and the difference shapes the rest of this module.

• A job (a command) says do this: "fetch the preview for x7Kp". It has one intended worker, and once it's done it can be forgotten. Job queues are built for this (07.05 A job queue on Postgres with SKIP LOCKED).
• An event says this happened: "link x7Kp was created". It has no idea who cares. The preview worker, the QR worker, the webhook sender and the search indexer can each react to it, and next year's new feature can too, without changing the code that creates links. Logs are built for this (07.02 Queues vs logs).

Events decouple the producer from its consumers; jobs make the intended work explicit. Snip uses both: link creation publishes one link.created event, and the services that care turn it into their own jobs.

A queue turns overload into delay

A queue doesn't make work cheaper. Workers still have to do every bit of it. What it changes is when. If links arrive faster than workers can fetch previews, the queue grows instead of requests failing. That's usually what you want for a burst, and a trap for a lasting overload.

Little's law connects the numbers: the average number of items waiting equals the arrival rate times the average wait. If previews arrive at 350 a second and each sits in the queue for 2 seconds on average, about 700 are waiting at any moment. Turn it round: a backlog of 70,000 at 350 a second means a new preview waits 200 seconds. Queue length alone tells you little; queue length divided by the drain rate tells you how late the work is.

So watch two numbers for every queue:

• Age of the oldest message: how late the work is. This is the one to alert on, because it's what users feel.
• Depth: how many messages are waiting. Useful for scaling workers, alarming only when it keeps growing.

And give every queue a limit. A queue that grows without bound eventually runs out of memory or disk, and then it can't accept anything at all, turning a slow consumer into an outage for the producer. When the limit is hit, shed or delay the least important work (03.11 Backpressure and load shedding).

What you give up

• Results arrive later, so the product has to show work in progress ("preview loading").
• Failures happen where nobody is watching. A request that fails shows the user an error; a background job that fails needs retries, alerts and a place for messages that keep failing (07.03 Delivery guarantees and idempotent consumers).
• Duplicates and ordering become your problem (07.03 Delivery guarantees and idempotent consumers, 07.04 Partitions, consumer groups and ordering).
• Tracing a single action across a request and five background steps needs a shared ID passed along with every message.

Recap

Interview questions