Where to put a rate limiter

The problem: 40 servers, one limit

Snip's limit is "100 link creations per minute per API key". But requests from one key are spread across 40 app servers by the load balancer. If each server keeps its own count, the key really gets 4,000 a minute. Where the counting happens, and what it's keyed on, matters as much as the algorithm.

The layers

Most systems use several. Snip limits by IP at the CDN (stopping floods of redirects or link creation from one address), by API key and plan in the gateway (100 link creations a minute on the basic plan), and inside the link service on the database's capacity (a cap on concurrent writes, 03.11 Backpressure and load shedding).

Sharing counts across servers

For a global "100 per minute per key", the count has to be shared. The usual store is Redis, with the check done atomically on the server so two app servers can't both read 99 and both allow:

• A fixed or sliding window counter: INCR the current window's key and set an expiry, in one script, compare with the limit.
• A token bucket: a short script that reads the bucket, refills it for the elapsed time, takes a token and writes it back, all as one atomic step.

That adds a network round trip (around half a millisecond, 01.04 Numbers to know) to every limited request, and Redis becomes a dependency of the API.

Local limits: no round trip

Alternatives trade precision for speed:

• Divide the limit. With 40 servers, each allows 100 ÷ 40 per key. No coordination, but uneven load balancing makes it inaccurate, and it changes whenever the fleet resizes.
• Local buckets, synced periodically. Each server counts locally and reconciles with a shared store every second or so. Limits are approximately right, overshooting briefly.

For abuse protection, approximate is fine; for billing quotas ("10,000 API calls a month"), use the shared, exact counter.

What to key on

• API key or user: the normal choice for authenticated APIs. Fair and hard to evade.
• IP address: for unauthenticated traffic. Many users can share one IP (offices, mobile carriers), and attackers can rotate through many, so set it generously and combine it with other signals.
• Route or cost: creating a link costs far more than reading one; weight requests by cost, or give expensive routes their own limit.
• Hot keys: one enormous customer's counter in Redis is a hot key (05.02 Hot partitions and celebrity keys); a local pre-check before the shared one keeps it off the shard.

When the limiter is down

If Redis is unreachable, should requests be allowed (fail open) or rejected (fail closed)? For most APIs, fail open: a brief window without per-key limits is better than rejecting every customer, and the service's own concurrency limits still protect it. Fail closed only where the limit protects something critical, such as a costly third-party quota.

Recap

Interview questions