Delayed and scheduled jobs

The problem: three kinds of "later"

Snip's paid plan has features that happen at a time rather than in response to a request:

The first instinct, a cron job on one server, works until that server is replaced, or there are two of them.

Delays and one-off times: store them as data

The Postgres queue from 07.05 A job queue on Postgres with SKIP LOCKED already supports this: a job with run_at in the future isn't claimed until then, and the partial index keeps such jobs cheap to skip.

-- Retry the webhook in 10 minutes
INSERT INTO jobs (queue, payload, run_at)
VALUES ('webhooks', '{"code": "x7Kp", "event": "link.created"}', now() + interval '10 minutes');

Hosted queues have limits here. Amazon SQS can delay a message by at most 15 minutes, either for the whole queue or per message, and suggests EventBridge Scheduler for anything longer. So a job that should run in three days needs a store that can hold future work, such as a database table, a scheduler service, or a sorted set in Redis scored by run time.

Don't make correctness depend on the job. Snip's link expiry looks like a scheduled job ("on Friday at 18:00, disable x7Kp"), but jobs run late, and a late expiry job would let a dead campaign link work for an hour. So the redirect itself checks expires_at on every request, using the copy in the cache. The scheduled job only tidies up: removing the link from the cache, notifying the owner. A job that's late now costs a little tidiness, not correctness.

Recurring schedules on many servers

The weekly report should run once a week. Snip runs its scheduler code on all three worker servers, for resilience, and each one wakes up on Monday at 09:00 and enqueues the report job. Now there are three reports.

Options:

• One designated scheduler, with a leader election so another takes over if it dies. Works, but leader election is its own topic (08.07 Leader election and leases).
• Let every scheduler try, and let the database pick one. Each schedule slot is a row with a primary key on (name, time). Every instance tries to insert it; exactly one succeeds and enqueues the job, in the same transaction:

export async function tick(db: pg.Pool, name: string, slot: Date, payload: object) {
  const client = await db.connect();
  try {
    await client.query("BEGIN");
    const won = await client.query(
      "INSERT INTO scheduled_runs (name, slot) VALUES ($1, $2) ON CONFLICT DO NOTHING RETURNING name",
      [name, slot],
    );
    if (won.rowCount === 1) {
      await client.query("INSERT INTO jobs (queue, payload) VALUES ($1, $2)", [name, payload]);
    }
    await client.query("COMMIT");
    return won.rowCount === 1;
  } catch (err) {
    await client.query("ROLLBACK");
    throw err;
  } finally {
    client.release();
  }
}

Tested with five scheduler instances ticking for the same slot at once: exactly one won, and exactly one job was queued. Because the slot is a fixed time (Monday 09:00), not "now", instances whose clocks differ by a few seconds still compete for the same row.

Missed runs

If every scheduler is down from Sunday night to Tuesday, Monday's report slot passes. When they come back, should it run? Decide per schedule:

• Catch up: run the missed slot now. Right for billing, cleanup and anything that must happen for each period.
• Skip: only run the next one. Right for "send the current dashboard snapshot", where a late copy is useless.
• Catch up, but only the latest: if ten slots were missed, run one, not ten.

Recording slots in a table, as above, makes this easy: on startup, look for slots between the last recorded one and now.

Time zones

"Monday at 09:00" means 09:00 in the owner's time zone, and that moves against UTC when clocks change for daylight saving. Store the schedule as a local time plus a named zone (Europe/London), not as a fixed UTC offset, and work out each next run in that zone. In zones that change their clocks, some local times happen twice a year and some don't happen at all, so don't schedule at 01:30 or 02:30 local time.

Recap

Interview questions