The problem: one event, many receivers
Three new Snip features all start from one event and end at many places:
- Several services must react to
link.created: previews, webhooks, search (07.01 Why move work off the request path). - Team feeds. On the team plan, each member's dashboard shows "recent links from your team". A team can have 5 members or, for one large customer, 40,000.
- Live counters. An agency's dashboard shows a link's clicks ticking up in real time. During a launch, 2,000 people watch the same viral link while it gets 50,000 clicks a second.
Each is a kind of fan-out: one input becoming many outputs. They need different tools.
1. Many services: publish once, subscribe many
Don't make the link service call each interested service; it would need to know them all, and a slow one would slow it down. Publish one event and let each service subscribe.
- With a log (07.02 Queues vs logs), each service is its own consumer group on the
link-eventstopic, reading every event at its own pace. - With queues, a topic copies each message into one queue per subscriber. Amazon's version is SNS in front of SQS: publish to an SNS topic, and each subscribed SQS queue receives its own copy, consumed and retried independently.
Either way, adding a fourth service means adding a subscriber, with no change to the publisher, and one service being down or slow doesn't hold back the others: its own queue or offset simply falls behind.
2. Many people's feeds: on write or on read
When Ana creates a link, how does it get into her teammates' "recent team links"?
Fan-out on write (push). A worker copies the new link into a feed list for each teammate as soon as it's created. Reading a feed is then one quick lookup of a precomputed list, the denormalised shape of 04.12 Denormalisation.
Fan-out on read (pull). Store nothing extra. When someone opens their dashboard, query "the newest links from everyone on my team" and merge them.
| On write | On read | |
|---|---|---|
| Cost of creating a link | One write per team member | One write |
| Cost of opening the feed | One lookup | A query across every teammate's links |
| Good when | Feeds are read far more often than written; groups are small or medium | Groups are huge, or most feeds are rarely read |
| Breaks when | One author has a huge audience: 40,000 writes per link | Reads are frequent and the merge is expensive |
The usual answer is hybrid: push to most teams, but for the few huge ones, don't fan out; merge their recent links in at read time. It's the same "celebrity" special case as hot keys (05.02 Hot partitions and celebrity keys), and it's how large social feeds are built (12.03 Feed generation: push, pull, hybrid).
3. Many live screens: aggregate, then push
The live counter is fan-out to connections: 2,000 browsers hold WebSocket connections
(03.05 Realtime: polling, SSE and WebSockets), spread over many gateway servers, all subscribed to x7Kp.
Pushing every click to every screen would be 50,000 × 2,000 = 100 million messages a second, for a number no human
can read that fast. So Snip aggregates first: one process per link sums its clicks for a second and publishes a
single x7Kp: 1,204,331 update. Then it fans out in two levels:
- The update goes to each gateway server that has at least one subscriber to
x7Kp: perhaps 20 servers. - Each gateway sends it down each of its own subscribed connections.
That's 20 messages between servers and 2,000 to browsers, once a second. Two rules keep it healthy:
- Each receiver gets its own buffer. A browser on a slow phone network mustn't slow the others. Give each connection a small outgoing buffer, and if it fills, drop the old updates, since only the latest count matters.
- Coalesce updates. If three updates are waiting for one slow connection, send only the newest. Numbers that replace each other don't need to arrive one by one.
Snip at the end of Module 07
5 steps. Use Next and Back to follow the request.
Text version
- Every redirect appends a click event to the clicks topic, keyed by code across 32 partitions (07.02, 07.04).
- The click counter applies batches and stores its offset in the same Postgres transaction as the counts, so redelivered batches are skipped (07.03).
- Per-second totals fan out to gateway servers and on to every dashboard watching the link (07.09).
- Creating a link writes the link and a link.created event to the outbox in one transaction (07.06).
- A relay publishes outbox events; services subscribe and turn them into jobs: previews, QR codes, webhooks with retries and backoff, scheduled reports (07.05, 07.07). Custom domains run as an orchestrated saga (07.08).
- Snip app → clicks topic: click
- Snip app → Postgres: link + event
- clicks topic → Click counter
- Click counter → Live counts: per second
- Postgres → link-events: relay
- link-events → Job workers: jobs
Nothing a user waits for now depends on a slow or outside system, every event is published exactly when its data commits, and every consumer is safe to run twice. Module 08 turns to what all of this has been quietly relying on: machines and networks that fail in partial, confusing ways (08.01 Partial failure).
Seen in production
Slack
2023Real-time messaging
Slack's channel servers each own a subset of channels, assigned by consistent hashing. Clients connect to gateway servers, which hold their WebSocket subscriptions. When a channel server receives a message, it sends it to every gateway server across the world that is subscribed to that channel, and each gateway sends it to every connected client subscribed to that channel. Slack says the system serves tens of millions of connected clients and delivers messages across the world in 500 ms.
What we foundTwo-level fan-out: a server per channel sends each message to every gateway with subscribers, and each gateway to its own connections.
Real-time Messaging (Slack Engineering)slack.engineering ↗Amazon
DocsSNS to SQS fan-out
When an SQS queue is subscribed to an SNS topic, every message published to the topic is sent to the queue. With several queues subscribed, one publish reaches each of them, and each service consumes and retries from its own queue independently of the others.
What we foundPublish once to a topic; each subscribed queue gets its own copy to process at its own pace.
Fanout Amazon SNS notifications to Amazon SQS queues for asynchronous processingdocs.aws.amazon.com ↗Snip's team feed uses fan-out on write. A 40,000-member customer's bot creates 100 links a minute. What happens, and what's the fix?
Recap
Interview questions
Fan-out on write or fan-out on read for a news feed? How do you handle users with millions of followers?