The outbox pattern and change data capture

The problem: the dual write

Snip publishes a link.created event to Kafka for the preview, webhook and search services (07.01 Why move work off the request path). The obvious code writes to two systems in a row:

Reversing the order is no better: publish first, and if the insert then fails, every service reacts to a link that doesn't exist. Retrying helps with timeouts but not with a process that died. This is the dual write problem: there's no transaction that spans Postgres and Kafka, so any code that writes to both can leave them disagreeing.

The outbox

The fix is to make it a single write. Alongside its normal tables, the database gets an outbox table, and the event is inserted into it in the same transaction as the change:

CREATE TABLE outbox (
  id           bigserial   PRIMARY KEY,
  topic        text        NOT NULL,
  key          text        NOT NULL,
  payload      jsonb       NOT NULL,
  created_at   timestamptz NOT NULL DEFAULT now(),
  published_at timestamptz
);
CREATE INDEX outbox_unpublished ON outbox (id) WHERE published_at IS NULL;

BEGIN;
INSERT INTO links (code, long_url, owner_id) VALUES ('x7Kp', 'https://shop.example/spring', 4821);
INSERT INTO outbox (topic, key, payload)
  VALUES ('link-events', 'x7Kp', '{"type": "link.created", "code": "x7Kp", "owner": 4821}');
COMMIT;

Either both rows commit or neither does. Then a separate relay copies committed outbox rows to Kafka. The microservices.io description of the pattern is honest about the catch: the relay "might publish a message more than once".

Relay 1: polling the table

export async function relayOnce(db: pg.Pool, publish: (topic: string, key: string, event: object) => Promise<void>) {
  const client = await db.connect();
  try {
    await client.query("BEGIN");
    const { rows } = await client.query(
      `SELECT id, topic, key, payload FROM outbox
        WHERE published_at IS NULL
        ORDER BY id LIMIT 100
        FOR UPDATE SKIP LOCKED`,
    );
    for (const r of rows) await publish(r.topic, r.key, { ...r.payload, eventId: r.id }); // may repeat after a crash
    await client.query("UPDATE outbox SET published_at = now() WHERE id = ANY($1)", [rows.map((r) => r.id)]);
    await client.query("COMMIT");
    return rows.length;
  } catch (err) {
    await client.query("ROLLBACK");
    throw err;
  } finally {
    client.release();
  }
}

Tested against Postgres with 250 links:

• A transaction that inserted a link and its outbox row, then rolled back, left neither.
• A relay that crashed partway through a batch of 100 republished that batch after restarting: 300 sends, 250 distinct events. Every event arrived; some arrived twice.
• Three relays running at once, without crashes, published each of the 250 events once, thanks to SKIP LOCKED.

So consumers must deduplicate, using the eventId the relay adds (07.03 Delivery guarantees and idempotent consumers). Mark old rows published and delete them after a few days, or the table grows for ever.

The ordering catch. bigserial IDs are handed out when rows are inserted, but transactions commit in their own order. Transaction A takes ID 10, transaction B takes ID 11 and commits first; a relay polling at that moment sees 11 but not 10, publishes 11, and publishes 10 on its next pass. For one key this is rare, since Snip's writes to one link are seconds apart, but consumers that care should compare a version number on the entity rather than trust arrival order.

Relay 2: change data capture

Postgres already writes every committed change, in commit order, to its write-ahead log (04.06 Write-ahead logs and crash recovery), and logical decoding can stream those changes out as rows. Change data capture (CDC) tools such as Debezium connect like a replica, through a replication slot, and turn each committed change into an event. Point one at the outbox table and it publishes every new outbox row, in commit order, without any polling. Debezium ships an "outbox event router" for exactly this.

CDC can also capture changes to ordinary tables, with no outbox at all. That's how the cache invalidator in 06.04 Invalidation and staleness learns about every edit, whoever made it. The outbox is still better for events other services depend on: it publishes a deliberate, stable event shape, not your internal table layout, which you'll want to change.

A replication slot keeps the WAL until the connector has read it. If the connector stops for a day, the WAL piles up on the primary's disk. Monitor slot lag the way you'd monitor replica lag.

Recap

Interview questions