Workflows and sagas

The problem: one action, four systems

Snip's paid plan adds custom domains: an owner can use go.shop.example/summer instead of snip.link/x7Kp. Setting one up touches four systems:

1. Charge the domain add-on to the owner's card, through a payment provider.
2. Record the domain in Snip's database as pending.
3. Get a TLS certificate for it from a certificate authority, which only works once the owner's DNS points at Snip.
4. Configure the CDN to serve the domain, then mark it active.

If step 3 fails because the owner hasn't changed their DNS, the card has been charged for a domain that doesn't work. In one database, a transaction would roll everything back. But the payment provider, the certificate authority and the CDN aren't in Snip's database, and none of them will join a transaction that spans the others.

Sagas

A saga is the answer from a 1987 paper by Hector Garcia-Molina and Kenneth Salem, written for long-running database transactions and now used across services. A long transaction is a saga if it can be written as a sequence of smaller transactions, each committed on its own, where the system guarantees that either all of them complete, or compensating transactions run to undo the ones that did.

For Snip's custom domain:

Compensation isn't rollback

A database rollback makes it as if the transaction never happened. A compensation is a new action that undoes the business effect, and the world can see both:

• Others may have seen the in-between state. Between steps 2 and 5, the dashboard showed "pending". That's a real state the product must design for.
• Some things can't be undone. An email can't be unsent; send it only after the last step that can fail, or send a follow-up instead. Order the steps so the hard-to-undo ones come last.
• Compensations can fail too. A refund call can time out. Compensations must be retried until they succeed, so, like every step, they must be safe to repeat (07.03 Delivery guarantees and idempotent consumers): refund with an idempotency key, so a retried refund doesn't refund twice.

Who drives the steps?

Orchestration. One component, the orchestrator, knows the steps, calls each service, records progress, and runs the compensations on failure. The whole process is in one place, readable and easy to monitor.

Choreography. No one is in charge. Payments publishes charge.succeeded; the domain service hears it and records the domain, then publishes domain.recorded; the certificate service hears that, and so on. Each service reacts to events (07.06 The outbox pattern and change data capture). It's loosely coupled, but the process exists only as the sum of many subscriptions, which makes it hard to see, change, or answer "where is this order stuck?".

A rough rule: choreography for simple reactions with two or three steps, orchestration once there's a real process with branches, waits and compensations. Snip's domain setup has a wait (for DNS that may take a day), so it's orchestrated.

type Step<C> = { name: string; run: (ctx: C) => Promise<void>; undo?: (ctx: C) => Promise<void> };

/** Run steps in order; if one fails, undo the completed ones in reverse, then rethrow. */
export async function runSaga<C>(steps: Step<C>[], ctx: C, record: (event: string) => Promise<void>) {
  const completed: Step<C>[] = [];
  for (const step of steps) {
    try {
      await step.run(ctx);
      await record(`done: ${step.name}`);
      completed.push(step);
    } catch (err) {
      await record(`failed: ${step.name} (${(err as Error).message})`);
      for (const done of completed.reverse()) {
        if (done.undo) {
          await done.undo(ctx); // must be retried until it succeeds, and safe to repeat
          await record(`undone: ${done.name}`);
        }
      }
      throw err;
    }
  }
}

Run with a certificate step that throws, it records done: charge card, done: reserve domain, failed: issue certificate, undone: reserve domain, undone: charge card, and leaves nothing charged or reserved. This sketch shows the shape, but it keeps its progress in memory: if the process dies between steps, nobody knows to carry on or compensate. A real orchestrator records each step durably (the record function writing to a table, in the same transaction as the step where possible) and resumes from that record after a crash, and it can wait for a day for DNS without holding anything open.

Durable workflow engines

That durable record-and-resume is hard to get right, so there are engines for it: Temporal, AWS Step Functions, Netflix's Conductor and others. In Temporal's model you write the workflow as ordinary code; every step's result is saved to an event history, and if the worker running it dies, another worker fetches the history and replays the code against it to rebuild its state and carry on. The price is that workflow code must be deterministic (no reading the clock or random numbers directly), because it's re-run during replay. Steps that talk to the outside world, called activities, are retried with backoff.

For a single process like Snip's domain setup, a jobs table and a saga table with each step's status (07.05 A job queue on Postgres with SKIP LOCKED) are enough. When a company has dozens of long processes with waits and branches, an engine pays for itself.

Recap

Interview questions