Notification fan-out

The problem: a creator with a million followers posts

A creator on Snip Bio posts a new link. Their 1.2 million followers who have notifications on should hear about it: a push notification on their phone, an update in the app if it's open, maybe a line in tomorrow's email digest. It should arrive within a minute or two, respect each person's settings and time zone, and never arrive twice.

Doing this inside the "publish link" request is impossible. It's a pipeline.

The pipeline

Expand in batches. A million recipients is a million small jobs. Paging through followers in batches of 1,000 creates 1,200 jobs, which spread across workers and can't overwhelm anything at once.

Decide per recipient. The most important step is often deciding not to send:

• Preferences: has this person muted this creator, or turned off "new link" notifications?
• Quiet hours: it's 03:00 where they are; hold it until morning, or move it to the digest.
• Rate limits: they follow 400 creators; ten posts in an hour should become one "10 new links from creators you follow", not ten buzzes.
• Deduplication: a key per (user, event), so a retried batch never notifies anyone twice (07.03 Delivery guarantees and idempotent consumers).

The channels

• In-app: if the user's app is connected (12.01 Millions of WebSocket connections), push the update over their connection. Cheap and instant, and it can replace a phone notification entirely.
• Mobile push: the app's servers send to Apple's and Google's push services, which deliver to the device. They have their own rules and limits, so a channel worker paces sends, retries with backoff, and removes device tokens the service reports as invalid.
• Email: batch into digests; individual emails for each event are rarely welcome.

Collapse what replaces itself

Some notifications supersede earlier ones: "your link has 1,000 clicks", then "2,000 clicks". The push services support this directly. With Google's Firebase Cloud Messaging, a collapsible message carries a collapse key, and a newer message with the same key replaces one not yet delivered: a phone that was offline gets only the latest count. FCM stores up to four different collapse keys per device at a time. Non-collapsible messages are each delivered individually.

Spread the spike

A million notifications released at once also means a million people opening the app at once. Stagger delivery over a couple of minutes, send to recently active users first (they're the likeliest to open it), and give the "notification opened" traffic path enough headroom.

Recap

Interview questions