Queues vs logs

The problem: one stream of clicks, many readers

Snip's click events started with one reader: the worker that adds them up (02.06 Async workers and queues). Now three more teams want them. Fraud detection wants to spot bots clicking a link thousands of times. Analytics wants every click in its data warehouse. And the paid plan wants to stream each owner's clicks to their own systems.

With a classic queue, a click message goes to one worker and is deleted once handled. Four readers would need four copies of every message, and a new reader next year would get nothing from the past. Snip needs a different shape of messaging for this. There are two, and choosing between them is one of the most common design decisions in this area.

A queue: each message to one worker, then gone

A message queue (Amazon SQS, RabbitMQ, the Postgres queue in 07.05 A job queue on Postgres with SKIP LOCKED) holds messages until a worker takes one, does the work and acknowledges it, after which the message is deleted.

• Workers compete. Ten workers share one queue, and each message goes to one of them. To go faster, add workers.
• Acknowledged one by one. If a worker dies holding a message, it comes back for another worker: in SQS when its visibility timeout runs out (30 seconds by default, up to 12 hours), in RabbitMQ when the worker's connection closes without an ack.
• No history. Once acknowledged, it's gone. A worker added tomorrow sees only tomorrow's messages.
• Order is loose. SQS standard queues make "a best-effort attempt" at order, and retried messages come back late.

That's exactly right for jobs: "fetch the preview for x7Kp" should be done once, by someone, and then forgotten.

A log: every event kept, in order, for every reader

A log (Apache Kafka, Amazon Kinesis, Redpanda) is an append-only sequence of events. Producers add to the end. Nothing is removed when it's read: in Kafka's words, "events are not deleted after consumption", and instead each topic keeps events for a configured time (say, 7 days) or size.

• Readers track their own position. Each reader remembers an offset: "I've processed everything up to event 18,422,001". Reading is cheap and doesn't change the log.
• Many independent readers. The click counter, fraud detection and the warehouse loader each have their own offset and go at their own pace. A new reader can start from the beginning of what's retained.
• Replay. Found a bug in the click counter? Fix it, reset its offset to yesterday, and recount.
• Ordered, per partition. A topic is split into partitions for scale, and order is kept within each one (07.04 Partitions, consumer groups and ordering).

That's right for events: "x7Kp was clicked from Germany at 14:02:11" is a fact that several readers will each use differently.

Side by side

The "one slow message" row is the trade-off people miss. Because a log reader's progress is a single number per partition, it can't move past an event it hasn't handled, so one bad message can stall everything behind it. 07.04 Partitions, consumer groups and ordering covers how to deal with that.

Choosing

• One consumer per message, work to be done, no need for history: a queue.
• Several consumers, or a likely future one, or the need to replay or reprocess: a log.
• Very high event rates (Snip's 100,000 clicks a second): a log, because appending sequentially and tracking one offset per partition is far cheaper than acknowledging each message.

Many systems use a log as the backbone and queues at the edges: a consumer reads events from the log and turns the ones that need work into jobs.

Recap

Interview questions