Millions of WebSocket connections

The problem: three million phones, always connected

Snip launches Snip Bio: public profile pages where creators list their links, and a mobile app where people follow creators and get live updates: new links, live click counters, "who's viewing your page right now". At peak, three million app users are online, each holding one WebSocket. A request-response API scales by requests per second; this scales by connections held at once, and connections are stateful.

What one server can hold

An idle WebSocket costs little CPU, but each one holds a socket (a file descriptor) and some memory for buffers and per-connection state. The first limits people hit are configuration, not hardware: the operating system's default limit of open files per process is often 1,024. Phoenix's team, benchmarking WebSockets in 2015, stopped at exactly 1,000 connections until they raised it, and eventually held 2 million idle connections on one 40-core, 128 GB server.

Real traffic is heavier than idle connections: messages to encode, TLS, heartbeats. Snip plans for 100,000 connections per gateway server, so three million users need about 30 servers plus headroom for a server or a zone to fail.

Two kinds of server

Split the job, as Slack and Discord do (07.09 Fan-out patterns):

• Gateway servers hold the connections: authenticate on connect, keep the socket alive, write messages out. They know nothing about business logic.
• Back-end services decide what to send to whom, and publish it.

The hard part is routing: an update for user 4821 must reach whichever gateway holds 4821's connection. Two common designs:

• Subscribe by topic. Each gateway subscribes, on a pub/sub system, to the topics its connected users need (user:4821, link:x7Kp). Publishers don't need to know where anyone is.
• A connection registry. When a user connects, record user 4821 → gateway 17 in a fast store with a TTL; senders look it up and send directly. Fewer subscriptions, but one more thing to keep accurate.

Snip uses topic subscriptions for shared topics (a link's live counter) and the registry for per-user messages.

Keeping connections healthy

• Heartbeats. Load balancers and mobile networks silently drop connections that look idle. Send a small ping every 20–30 seconds in each direction, and treat a missed reply as a dead connection, on both sides.
• Resume, don't restart. A phone going through a tunnel disconnects for ten seconds. If the server numbers each message to a user, the client reconnects with "last seen #1,842" and gets only what it missed, rather than reloading everything.
• Per-connection buffers. A slow client must never block the others. Give each connection a bounded outgoing buffer; if it fills, drop old updates that have been superseded (a counter), or disconnect the client so it resumes later (07.09 Fan-out patterns).

Deploys without a reconnect storm

Restarting a gateway closes 100,000 connections, and all of them reconnect within a second. Multiplied across a rolling deploy, that's a self-inflicted surge. Defences:

• Drain slowly. Tell the load balancer to stop sending new connections to the old server, then close existing ones gradually over minutes, not all at once.
• Jittered reconnects. Clients wait a random delay before reconnecting, growing on repeated failures (03.09 Timeouts, retries, backoff and jitter).
• Resume tokens, so reconnecting is cheap for the back end, not a full reload.

Recap

Interview questions