Circuit breakers and bulkheads

The problem: a slow safety service stops redirects

Snip's link-creation API calls the outside URL-safety service (02.06 Async workers and queues) on a background worker, but the paid plan added a synchronous "check before saving" option. One afternoon the safety service slows from 200 ms to 30 seconds per request. Each app server has a pool of 50 outbound HTTP connections, shared by every dependency. Within a minute, all 50 are stuck waiting on the safety service. Now calls to the payment provider and to the internal links service also wait for a connection. Redirects, which never touch the safety service, start failing.

A failure in one dependency has spread to everything, through a shared resource. Two patterns stop that.

Circuit breakers: stop calling what's broken

A circuit breaker wraps calls to one dependency and counts failures. Martin Fowler's description has three states:

• Closed (normal): calls go through. Failures are counted.
• Open: after a threshold of failures, the breaker trips and every call fails immediately, without touching the dependency, for a cool-down period.
• Half-open: after the cool-down, a trial call goes through. If it succeeds, the breaker closes; if it fails, it opens again for another cool-down.

type BreakerState = "closed" | "open" | "half-open";

export class CircuitBreaker {
  state: BreakerState = "closed";
  private failures = 0;
  private openedAt = 0;

  constructor(
    private readonly threshold = 5, // consecutive failures that trip it
    private readonly coolDownMs = 30_000, // how long to fail fast before trying again
    private readonly now: () => number = Date.now,
  ) {}

  async call<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === "open") {
      if (this.now() - this.openedAt < this.coolDownMs) throw new Error("circuit open: failing fast");
      this.state = "half-open"; // let one trial call through
    }
    try {
      const result = await fn();
      this.state = "closed";
      this.failures = 0;
      return result;
    } catch (err) {
      this.failures++;
      if (this.state === "half-open" || this.failures >= this.threshold) {
        this.state = "open";
        this.openedAt = this.now();
      }
      throw err;
    }
  }
}

Tested with a threshold of 3 and a 1-second cool-down: three failing calls opened the breaker, the next two failed fast without calling the dependency, a call at 500 ms still failed fast, a trial at 1.2 s failed and reopened it, and a trial after the next cool-down succeeded and closed it. (This sketch lets every call through while half-open; real libraries admit only one or a few trial calls.)

What a breaker buys:

• The caller stops wasting time and connections on a dependency that's down, and its own users get a fast error (or a fallback) instead of a 30-second wait.
• The dependency gets room to recover instead of being hammered by retries.

What to do when it's open: a fallback. Snip's "check before saving" falls back to saving the link and queuing the check, as it did before the paid feature existed. A breaker without a sensible fallback just turns slow errors into fast ones, which is still an improvement.

Tune it on real numbers: trip on a failure rate over a window (say, over 50% of the last 20 calls, counting timeouts as failures), not on one bad call; keep the cool-down short enough that recovery is noticed quickly.

Bulkheads: don't share what one failure can exhaust

A ship's hull is divided into watertight compartments, so a breach floods one section rather than sinking the ship. The bulkhead pattern does the same with resources: give each dependency its own pool of connections, threads or concurrency slots, so one that hangs can exhaust only its own.

Snip splits each app server's outbound connections:

With bulkheads, the safety service's slowdown fills only its 5 connections. Further calls to it wait briefly for a slot and then fail fast, the breaker trips, and redirects never notice. The same idea applies at every level: separate worker pools for different queues (07.05 A job queue on Postgres with SKIP LOCKED), separate database connection pools for the API and for batch jobs, separate clusters for the free and paid plans.

Where they fit

Timeouts first (03.09 Timeouts, retries, backoff and jitter), then bulkheads to contain, then breakers to fail fast and give the dependency a rest, then load shedding at the front door (03.11 Backpressure and load shedding). Each one is a new mode of behaviour, so test it: break a dependency on purpose in staging and watch the breaker open and the fallback work.

Recap

Interview questions