Forward and reverse proxies
A proxy is a server that passes requests along on someone's behalf. Which side it works for gives it its name:
- A forward proxy works for clients. A company's office proxy sends employees' requests out to the internet, filtering or logging them on the way. The websites see the proxy, not the employees.
- A reverse proxy works for servers. Clients send requests to it thinking it's the site; it passes them to the real servers behind it. The clients see the proxy, not the servers.
Every load balancer from 03.06 Load balancers that reads HTTP is a reverse proxy. Common software for the job includes nginx, HAProxy and Envoy, and every cloud provider sells a managed one.
Jobs a reverse proxy does well
Because every request passes through it, a reverse proxy is a natural home for work that's the same for every request, so each application doesn't have to do it:
| Job | What it means |
|---|---|
| TLS termination | Decrypt once at the edge; certificates live in one place. |
| Load balancing and health checks | 03.06 Load balancers. |
| Compression | Gzip or Brotli responses on the way out. |
| Caching | Keep copies of cacheable responses, like a small CDN in your own datacenter. |
| Connection handling | Hold thousands of slow client connections and keep a small pool of fast ones to the app. |
| Request limits | Reject huge bodies, slow uploads and malformed requests before they reach the app. |
| Request IDs and logs | Tag each request with an ID and log it uniformly. |
API gateways
An API gateway is a reverse proxy that also handles the cross-cutting jobs of an API:
- Authentication: check the API key or token once, and pass the caller's identity to the services behind.
- Rate limiting: each key gets so many requests a minute; the rest get
429(designed fully in 14.03 Rate limiting algorithms). - Routing to services:
/links/*to the links service,/stats/*to the stats service, so clients see one API even when there are many services behind it. - Request shaping: version routing, adding or stripping headers, sometimes combining several calls into one.
- Usage metering: counting calls per key for billing and dashboards.
3 steps. Use Next and Back to follow the request.
Text version
- A developer's request arrives with an API key. The gateway terminates TLS and assigns a request ID.
- It looks the key up (cached in memory) to find the account, and checks the account's rate limit. Over the limit means a 429 here, before any service does work.
- It routes by path. The services receive a verified account ID in a header, so none of them parse API keys.
- Developers → API gateway
- API gateway → Key store: check key
- API gateway → Links service: /links/*
- API gateway → Stats service: /stats/*
What doesn't belong in the gateway
A gateway sees every request, so it's tempting to put logic there. Resist:
- Business logic. "Free accounts can't create custom codes" belongs in the links service. Logic in the gateway is hidden from the team that owns the feature and is shared by every route, so every change is risky.
- Slow work. Everything the gateway does adds latency to every request. Keep checks to in-memory lookups.
- Being a single point of failure. Run several instances behind a layer 4 or managed load balancer, like any other stateless tier.
A good rule: the gateway handles who you are and how much you may call; services handle what you may do.
Seen in production
Cloudflare
2022Pingora
Cloudflare replaced NGINX with Pingora, its own proxy in Rust, serving over a trillion requests a day. A key reason was connection reuse: NGINX's connection pools are per worker process, so connections to origins were scattered and reused less. Pingora shares pools across threads. Cloudflare reported about 70% less CPU and 67% less memory for the same traffic, a third as many new connections per second, and a 5 ms lower median time to first byte.
What we foundAt reverse-proxy scale, connection reuse is a big lever: sharing connection pools cut new connections, CPU and latency.
How we built Pingora, the proxy that connects Cloudflare to the Internetblog.cloudflare.com ↗Snip's links service and stats service both check API keys and rate limits, with slightly different code. What's the main benefit of moving that into an API gateway?
Recap
Interview questions
What is an API gateway, and what would you put in it?