The problem: "add a cache" isn't one decision
By the end of Module 05, Snip already has caches in three places: a CDN for its static files, a tiny in-process cache on each app server for viral codes, and a Redis cluster for every redirect. They weren't planned as a set; each was added when something hurt. Now the team wants to know: where should a copy of a link live, and why there and not somewhere else?
Follow one redirect for snip.link/x7Kp from a phone in Sydney. Every hop it passes is a place that could answer it:
6 steps. Use Next and Back to follow the request.
Text version
- The browser keeps responses it's allowed to (by their Cache-Control headers). A repeat click answered here costs nothing at all.
- A CDN edge in Sydney can answer for every user in the city: one copy, shared by thousands, a few milliseconds away.
- A reverse proxy in front of the app (03.07) can cache whole responses too, shared by every user who reaches that datacenter.
- Inside the app, a small map in memory answers in about 100 ns, but only for the requests that land on that one server.
- Redis is shared by all the app servers: one network round trip (about 0.5 ms) and the same answer whichever server asks.
- Even the database caches: Postgres keeps recently used pages in memory, and the operating system keeps more. A 'database read' is often a memory read.
- Browser → CDN edge
- CDN edge → Reverse proxy
- Reverse proxy → Snip app
- Snip app → Redis: miss
- Snip app → Postgres: miss
The layers, compared
| Layer | Typical cost of a hit | Shared by | Who can clear it |
|---|---|---|---|
| Browser | Nothing: no request is sent | One user | Only expiry: you can't reach into someone's browser |
| CDN edge | A few ms, nearby | Everyone near that edge | You, by purging, or expiry |
| Reverse proxy | Under a ms, inside the datacenter | Everyone reaching that datacenter | You |
| In-process (app memory) | About 100 ns | Requests on one app server | Each server separately |
| Shared cache (Redis, Memcached) | About 0.5 ms round trip | Every app server | You, with one delete |
| Database buffer cache | A query's CPU, no disk | Every query | The database, automatically |
Two things change as you move outwards, towards the user. Hits get cheaper and save more work, because a hit at the edge skips everything behind it. But you get less control: a copy in a browser stays until it expires, whatever you do. That's why the outer layers suit data that's the same for everyone and rarely changes (fingerprinted files, 02.05 A CDN for static content), and the inner layers suit data that changes or is personal.
Local or shared?
The biggest design choice inside the datacenter is whether the app keeps its own copy in memory or asks a shared cache.
| In-process cache | Shared cache (Redis) | |
|---|---|---|
| Speed | Fastest: no network | One round trip |
| Hit rate | Each server warms its own copy, so lower | One copy serves everyone, so higher |
| After a deploy | Every new server starts empty | Stays warm |
| Consistency | 40 servers, 40 copies that can disagree | One copy |
| Memory | Takes memory from the app | Its own machines |
The common answer is both, as two levels: a small in-process cache with a short TTL (seconds) for the hottest keys, in front of a large shared cache with a longer one. The in-process level absorbs viral traffic (05.02 Hot partitions and celebrity keys); the shared level gives a high hit rate and survives deploys. The short TTL bounds how long the 40 local copies can disagree.
The danger: a cache you can't live without
A cache that's down should make Snip slower, not stop it. But at 100,000 redirects a second with a 99% hit rate, the database sees 1,000 reads a second, and it was sized for that. If Redis disappears, it sees 100,000, a hundred times its normal load, and falls over. The cache has stopped being an optimisation and become part of the capacity.
Ways to stay honest about it:
- Know the number. Work out what the database would see with the cache empty, and decide whether you can survive it.
- Protect the database anyway. Limit how many cache misses can reach it at once (03.11 Backpressure and load shedding), and coalesce them (06.05 Stampedes and request coalescing), so a cold cache means slow requests rather than a dead database.
- Warm before serving. Bring new cache nodes or regions into use gradually, filled from a warm copy.
- Test it. Turn the cache off in a test environment, or empty one node in production, and watch what happens.
Seen in production
Amazon
2019Service caches
The article compares in-memory caches inside each service with external caches such as Memcached or Redis. In-memory caches suffer from cold starts: a new server begins empty and sends a burst of requests to the service behind it. It describes a service that "has become addicted to its cache", where behaviour differs sharply depending on whether an item is cached, and recommends planning for the cache being empty. It also covers caching error responses with their own TTL, and keeping a soft TTL and a hard TTL, so stale data can be served when the service behind the cache is down.
What we foundLocal caches start cold on every new server, and a service that has grown used to its cache can fall over without it.
Caching challenges and strategies (Jas Chhabra, Amazon Builders' Library)builder.aws.com ↗Snip deploys 40 new app servers every afternoon. Right after each deploy, the database's read load triples for a minute. Which cache is the likely cause?
Recap
Interview questions
Where can you cache in a web system, and how do you decide where to put a cache?