Observability: logs, metrics, traces

The problem: "redirects are slow" at 14:07

An owner reports that Snip links "hang for a second" since lunchtime. Snip has 40 app servers, Redis, Postgres shards, a CDN and a dozen services. Somewhere among them, something changed. The team needs to answer three questions, fast: is something wrong (and for whom), where, and why.

Each question is best answered by a different kind of telemetry.

Metrics: is something wrong?

Metrics are numbers over time: requests per second, error rate, p99 latency, queue depth, CPU. They're cheap to store (one number per series per interval, however much traffic there is), fast to query over months, and what alerts are built on.

Google's SRE book names the four golden signals to measure for any user-facing service:

• Latency: how long requests take. Track successful and failed requests separately: a fast error and a slow success mean different things.
• Traffic: demand, in the service's own terms (redirects per second).
• Errors: the rate of failing requests, whether explicit (500s), implicit (wrong content) or by policy (too slow).
• Saturation: how "full" the service is: the resource closest to its limit.

The cost to watch is cardinality: every distinct combination of labels is a separate series. latency{route, status} is fine; latency{link_code} across 600 million links is 600 million series and will sink the metrics system. Per-link numbers belong in the analytics pipeline (11.01 Counting at scale), not in operational metrics.

Logs: what happened, exactly?

Logs are records of individual events: this request, this error, this retry. They have all the detail metrics throw away, so they answer "which requests?" and "what did the error say?".

• Structured logs (JSON with fields such as route, status, duration_ms, trace_id) can be filtered and counted; free text can only be searched.
• Volume is the cost. At 100,000 redirects a second, a log line per redirect is billions a day. Log every error and every slow request, and sample the rest.
• Never log secrets or personal data you don't need: passwords, tokens, full signed URLs (13.05 Signed URLs and access control).

Traces: where did the time go?

A request for x7Kp passes through the load balancer, an app server, Redis, maybe a Postgres shard. Distributed tracing follows it: each step is a span with a start time, a duration and attributes, and spans link into a tree by passing a trace ID (and the parent span's ID) along with every call. Google's Dapper paper, the ancestor of most tracing systems, describes exactly this: a trace tree of spans, with context propagated through common libraries.

Tracing everything is expensive, so traces are sampled. Dapper found that even sampling 1 in 1,024 requests still gave enough data for high-volume services. Many systems also keep every trace that errored or was unusually slow (tail-based sampling), since those are the ones you'll want.

Today, OpenTelemetry is the common standard for producing all three signals, with spans, attributes and context propagation defined the same way across languages and vendors.

How they fit together during an incident

1. A metric alert fires: p99 redirect latency over 200 ms for 5 minutes.
2. The dashboard narrows it: only redirects that miss the cache; only on shard 7.
3. An exemplar trace from the slow requests shows 900 ms in one Postgres query.
4. That trace's ID finds the logs: the query plan changed after a deploy at 13:58 dropped an index.

Linking them is what makes them powerful: every log line and metric exemplar carries the trace ID.

Recap

Interview questions