A function call that crosses a network
As Snip grows, its team splits the app into services: a links service that owns links, a redirect service on the hot path, and a stats service that owns counts. Now, when the dashboard needs a link and its stats, one service calls another over the network.
It looks like a function call. It isn't. A local call either returns or throws. A network call goes through many steps, and any of them can fail: the request may not arrive, the server may crash halfway, the reply may be lost. The nastiest case is when the caller hears nothing at all:
5 messages. Use Next and Back to follow them in order.
Text version
- Dashboard API → Links service: DeleteLink(x7Kp) — The dashboard asks the links service to delete a link, and starts a 1-second timer.
- Links service → Its database: DELETE x7Kp — The links service deletes the row. The work is done.
- Its database → Links service: ok (reply) — The database confirms.
- Links service → Dashboard API: ok (reply) (fails) — The reply is lost: a network blip, or the links service is paused for garbage collection past the timeout.
- Note over Dashboard API: 1 second passes: timeout
- Dashboard API → Dashboard API: unknown outcome — The caller can't tell 'never happened' from 'happened, reply lost'. It must decide what to do without knowing. Retrying a delete is harmless; retrying a 'create' or a 'charge' might not be.
Every design for service calls has to answer: what do we do when the outcome is unknown? The next two lessons (03.09 Timeouts, retries, backoff and jitter and 03.10 Idempotency keys) are the standard answers: retry carefully, and make retries safe.
Sync or async?
There are two basic ways for one service to involve another:
- Synchronous request–response (HTTP, gRPC): call and wait for the answer. Simple to reason about, and the only option when you need the answer to continue. But the caller is now only as available and as fast as everything it calls.
- Asynchronous messages (a queue or event log, 02.06 Async workers and queues): publish "link x7Kp was deleted" and carry on. Whoever cares (stats, search, billing) reacts in their own time. The publisher doesn't depend on them being up, at the cost of results arriving later.
A useful default: ask synchronously when you need the answer now; announce asynchronously when others need to know something happened. Snip's dashboard asks the links service for a link (sync). When a link is deleted, the links service publishes an event, and stats and the redirect cache react to it (async).
Chains multiply
Each synchronous hop adds its latency and its failure chance:
| Calls in a chain, each 99.9% available and 5 ms at p50 | Availability | Typical latency |
|---|---|---|
| 1 | 99.9% | 5 ms |
| 3 | 99.7% | 15 ms |
| 10 | 99.0% | 50 ms |
And the slowest call in any chain sets the pace (03.12 Tail latency and why p99 matters). So keep synchronous chains short: fetch in parallel where you can, cache answers that rarely change, and move anything that doesn't need an answer to async events.
Finding each other
Services run on many machines that come and go. A caller needs to find a healthy instance:
- DNS or a virtual address: call
links.internal, which resolves to a load balancer or a set of instances. Simple; updates are only as fast as DNS caching allows. - A service registry: instances register themselves and send heartbeats; callers (or their proxies) ask the registry for the current list and balance across it themselves.
- A service mesh: a small proxy runs next to every service instance (a "sidecar") and handles discovery, load
balancing, retries, timeouts, encryption between services and metrics, so each service just calls
localhost. It gives consistent behaviour everywhere, at the cost of an extra hop per call and a lot of infrastructure to run.
Seen in production
Amazon
2019Builders' Library
Amazon's article breaks a single request and reply into eight steps: the client posts the request, the network delivers it, the server validates it and updates its state, then posts a reply, which is delivered, validated and acted on by the client. Each step can fail independently. When the client times out, the result is "UNKNOWN": the request may have succeeded, failed, or been received but not processed, and the client must handle that case correctly.
What we foundOne request–reply has eight steps that can each fail, and a timeout leaves the result unknown, which code must handle.
Challenges with distributed systemsaws.amazon.com ↗When a Snip link is deleted, the stats service should stop showing it and the redirect cache should drop it. The delete request shouldn't fail just because stats is down. How should the links service tell them?
Recap
Interview questions
When should one service call another synchronously, and when should it use events?