The problem: every machine has its own time
Each server's clock is a quartz crystal, and no two run at exactly the same speed. Left alone, they drift apart by seconds a week. NTP keeps them close by checking against time servers and correcting, but "close" across a data centre usually means within a few milliseconds, and much worse if NTP is misconfigured, a VM is paused, or a network is congested.
Corrections cause their own trouble. NTP can slew a clock (run it slightly fast or slow until it's right) or step it (jump it, sometimes backwards). Leap seconds have historically been handled by repeating a second. So a program that reads the time twice can see the second reading come out earlier than the first.
Two clocks in every machine
Operating systems offer two kinds of clock, for two different questions:
| Wall clock ("time of day") | Monotonic clock | |
|---|---|---|
| Answers | What time is it? | How much time has passed? |
| Can jump backwards | Yes, when corrected | No, never |
| Comparable across machines | Roughly, within the skew | No: it's just a counter since boot |
| Use it for | Timestamps people read, Snowflake's time field | Timeouts, latencies, rate limits |
Measuring a duration with the wall clock is a classic bug: a backwards step makes it negative, and code that never expected a negative duration breaks, as Cloudflare found below.
Ordering events across machines
Timestamps from two machines can only order events that are further apart than the clocks' disagreement. Two Snip servers with clocks 5 ms apart can produce Snowflake IDs that sort in the "wrong" order for events less than 5 ms apart. For sorting a dashboard, that's harmless. For deciding which write wins, it isn't:
3 messages. Use Next and Back to follow them in order.
Text version
- Server A (+40 ms) → Database: set URL = old.example @ 10:00:00.040 — Ana edits a link's destination. Server A's clock is 40 ms fast, so the write is stamped 10:00:00.040.
- Server B → Database: set URL = new.example @ 10:00:00.020 — 10 ms later in real time, Ana fixes a typo. Server B's clock is correct and stamps it 10:00:00.020.
- Database → Server B: keep old.example (reply) (fails) — Last write wins compares timestamps: .040 beats .020, so the newer edit is silently thrown away. No error, no conflict, just lost data.
Last write wins is popular because it's simple, and many distributed databases use it to settle concurrent updates. Its cost is that a skewed clock silently discards writes. Use it only where losing one of two near-simultaneous writes is acceptable.
Ordering without trusting the clock
- Let one place decide. If every write to a row goes to that row's single primary (04.07 Replication: leaders, followers and lag), the primary's order is the order. This is why sharding with one leader per shard sidesteps most clock problems.
- Version numbers. Store a version on the row, and make each update conditional:
UPDATE links SET long_url = $1, version = 8 WHERE code = 'x7Kp' AND version = 7. If someone else got there first, zero rows change and the app retries with fresh data. Order comes from the data, not the clock. - Logical clocks. A Lamport clock is a counter each machine increments on every event and includes in every message; a machine receiving a larger counter jumps its own past it. It orders causes before effects without any physical time. Hybrid logical clocks combine a physical timestamp with a counter, staying close to real time while never going backwards. Part 2 covers these in depth (08.05 Clocks and ordering).
- Bound the uncertainty. Google's Spanner gives up on exact time and asks for an interval guaranteed to contain it, then waits out the uncertainty before committing (below).
What Snip does
- Durations, timeouts and rate limits use the monotonic clock.
- Snowflake-style click event IDs are generated with the wall clock; the generator refuses if time goes backwards (05.05 Distributed ID generation), and slightly misordered clicks across servers don't matter for statistics.
- Link edits go to the link's shard primary, and the dashboard sends the version it loaded, so a stale edit fails instead of overwriting a newer one.
- NTP is monitored: an alert fires if any server's offset passes 100 ms.
Seen in production
Cloudflare
2017RRDNS
Cloudflare's DNS software measured upstream round-trip times by subtracting two wall-clock readings, and Go's time function wasn't monotonic at the time. At the leap second on 1 January 2017 the clock stepped back, some durations went negative, and the server selection code passed them to a random number function that panics on negative input. About 0.2% of DNS queries were affected at peak. The fix made the software forget its upstream measurements whenever time jumped backwards.
What we foundA wall clock stepping back one second made a duration negative, and code that assumed time never goes backwards failed.
How and why the leap second affected Cloudflare DNSblog.cloudflare.com ↗Spanner TrueTime
Spanner's TrueTime API returns not a time but an interval, earliest to latest, guaranteed to contain the true time. It's backed by GPS receivers and atomic clocks in every data centre, and in Google's production environment the uncertainty was typically between 1 and 7 ms. Before a transaction's commit becomes visible, Spanner waits until its timestamp is certainly in the past (commit wait), which lets timestamps order transactions globally.
What we foundInstead of pretending clocks agree, return an interval with a guaranteed bound, and wait out the uncertainty.
Spanner: Google's Globally-Distributed Database (OSDI 2012)static.googleusercontent.com ↗A rate limiter measures how long since a user's last request with `now() - last`, using the wall clock. What can go wrong?
Recap
Interview questions
Why can't you rely on timestamps to order events in a distributed system?