The problem: a bug in last month's numbers
A bug in Snip's click consumer counted clicks from link previews (chat apps fetching a link to show a thumbnail) as real clicks. It ran for three weeks. Owners' charts, totals, unique visitors and top referrers are all inflated by different amounts. If the only thing Snip stored was the counts, the true numbers would be gone for good.
Raw events are the truth; everything else is derived
Separate two kinds of data:
- Raw events: every click as it happened (code, time, country, referrer, user agent, visitor hash). Append-only, never edited. Large, but cheap to keep compressed in object storage (09.04 How object storage works), partitioned by day.
- Derived data: everything computed from them. Per-minute counts, daily unique-visitor sketches, top referrers, latency histograms. Small, fast to query, and replaceable.
Because derived data can always be recomputed from raw events, a bug in a derivation is fixable: correct the code, recompute the affected period, swap the results in. Snip fixed the preview bug by filtering previews by user agent and recomputing three weeks of rollups from the archive overnight.
Rollups
A rollup pre-aggregates raw events at a coarser grain: one row per (link, country, minute) instead of one per click. Analytics stores do this at ingestion; Apache Druid, for example, combines rows with identical dimensions and truncated timestamps into one, which can cut row counts by orders of magnitude. The documentation is clear about the cost: you lose the ability to query individual events.
So choose rollup dimensions deliberately, and keep the raw events somewhere, so a question nobody anticipated ("clicks from Android in Brazil on Tuesday") can still be answered, slowly, from the archive.
Snip's tiers:
| Data | Kept for | Where |
|---|---|---|
| Raw click events | 13 months | Compressed daily files in object storage |
| Per-minute rollups | 30 days | Analytics database |
| Per-hour rollups and hourly HyperLogLog sketches | 2 years | Analytics database |
| Per-day totals | For the link's lifetime | Postgres |
Batch or stream?
- Batch processing runs over a bounded chunk of data (yesterday's files), producing complete, correct results, late.
- Stream processing handles events as they arrive (11.02 Time-window aggregation), producing fresh results that may need correcting as late data turns up.
The Lambda architecture runs both: a stream path for fresh, approximate numbers and a batch path that periodically recomputes correct ones and overwrites them. Jay Kreps pointed out its main cost: you implement your logic twice, in two different systems, and keeping code that must produce the same result in two complex distributed systems in sync is really hard.
His alternative, sometimes called the Kappa architecture: keep the raw events in a log (or an archive that can be replayed as one), run one stream processing job, and when the logic changes, start a second instance of the new version from the beginning of the retained data, let it catch up, and switch readers to its output. One code path; a recompute is a replay.
Snip uses that shape: one streaming job produces the rollups; the raw archive can be replayed through a new version of the same job to rebuild any period. A nightly batch job still exists, but runs the same code over the archive, only to reconcile and catch the stragglers that arrived too late for the stream (11.02 Time-window aggregation).
Snip at the end of Module 11
5 steps. Use Next and Back to follow the request.
Text version
- App servers pre-aggregate clicks per code for one second and append them to the clicks topic (11.01).
- A stream job counts by event time in one-minute windows with a watermark, builds HyperLogLog and count-min sketches, and handles late events (11.02 to 11.04).
- Rollups go to the analytics database for charts; live counters and visitor sketches go to Redis.
- Every raw event is also archived as compressed hourly files: the source of truth for every derived number.
- When logic changes or a bug is found, the same job replays the archive to rebuild any period (11.06).
- Snip app → clicks topic: click counts
- clicks topic → Stream job
- clicks topic → Raw archive: hourly files
- Stream job → Analytics DB: per-minute rows
- Stream job → Redis
- Raw archive → Stream job: replay
Every number Snip shows can now be traced to raw events and recomputed. Module 12 turns to the parts of Snip that are about people and places: live connections, feeds and location (12.01 Millions of WebSocket connections).
Seen in production
Jay Kreps
2014Questioning the Lambda Architecture
Kreps describes the Lambda Architecture as feeding an immutable sequence of records into a batch system and a stream processing system in parallel, implementing the transformation logic twice. His main criticism is the burden of maintaining code that must produce the same result in two complex distributed systems. As an alternative, he suggests starting a second instance of the stream job from the beginning of the retained data and, once it has caught up, switching the application to read from its output, adding that this "may be too simple of an idea to merit a Greek letter".
What we foundWriting the same logic twice for batch and stream is the real cost; reprocess by replaying the log through one job instead.
Questioning the Lambda Architecture (O'Reilly Radar)oreilly.com ↗Apache Druid
DocsRollup
Rollup is a form of summarization or pre-aggregation: Druid combines into a single row any rows with identical dimension values and timestamps after truncation to the query granularity. It's enabled by default and can reduce row counts by potentially orders of magnitude, but as a trade-off you lose the ability to query individual events.
What we foundPre-aggregate at ingestion to cut rows by orders of magnitude, at the cost of individual events.
Data rollup (Apache Druid documentation)druid.apache.org ↗Snip only stores per-hour rollups by (link, country). Product wants 'clicks by device type for last quarter'. What's possible?
Recap
Interview questions
Design an analytics pipeline for click events. How do you handle a bug that corrupted a month of metrics?