Checksums and silent corruption

The problem: a correct-looking wrong answer

An owner downloads last March's click export from Snip and opens it. One row reads x7Kp,2026-03-14,DE,… with a few characters of nonsense in the middle. No error was logged anywhere: the disk returned the bytes, the network delivered them, the app streamed them. Somewhere along the way, a few bits flipped.

That's silent data corruption: the hardware or software reports success and hands back the wrong data. It's rare per operation, and at Snip's scale (billions of reads, petabytes stored) rare things happen weekly.

Where bits go wrong

• Disks: a sector that decays over time ("bit rot"), a write that lands in the wrong place, firmware bugs.
• Networks: corruption that slips past link-level checks. TCP's 16-bit checksum is weak, and some errors pass it.
• Memory: a flipped bit in RAM without error-correcting (ECC) memory.
• Software: a bug that writes the wrong buffer, or reads a page before it's fully written.
• CPUs: an occasional core that computes wrong results. Google and Facebook both found such cores in their fleets (below), and they produce no error at all.

Redundancy (09.05 Replication vs erasure coding) doesn't help by itself. If a replica silently returns wrong data, you serve it. You need to detect first.

Checksums

A checksum is a short value computed from data, such as CRC32C, CRC64 or a cryptographic hash like SHA-256. Store it with the data; recompute it on read; if they differ, the data changed. A good checksum makes it overwhelmingly unlikely that random corruption leaves it unchanged.

import { crc32 } from "node:zlib";

const BLOCK = 4096;

export function writeWithChecksums(data: Buffer) {
  const blocks: { data: Buffer; crc: number }[] = [];
  for (let i = 0; i < data.length; i += BLOCK) {
    const block = data.subarray(i, i + BLOCK);
    blocks.push({ data: Buffer.from(block), crc: crc32(block) });
  }
  return blocks;
}

export function readVerified(blocks: { data: Buffer; crc: number }[]) {
  return blocks.map((b, i) => {
    if (crc32(b.data) !== b.crc) throw new Error(`block ${i} is corrupt: checksum mismatch`);
    return b.data;
  });
}

Tested on 12 KB of data in four blocks: reads verified cleanly, then flipping a single bit in block 2 made the next read fail with "block 2 is corrupt", instead of returning wrong data.

Where to check

Per block, on read. Storage systems keep a checksum per block or page and verify it every time it's read. In Postgres, each data page carries a checksum updated when it's written and verified each time it's read; current versions enable this by default. ZFS checksums every block, and on a mismatch repairs it automatically from a redundant copy.

End to end. The strongest check is computed where the data is created and verified where it's finally used, covering every hop in between. Snip's export worker computes a CRC64 of each export as it writes it and sends it with the upload; the object store verifies it on arrival, stores it, and the download can be checked against it. S3 accepts several checksum algorithms on upload, with CRC64NVME as the default, and AWS's own SDKs calculate one on every upload.

In the background (scrubbing). Data that's rarely read, like Snip's archives, could rot for years unnoticed and be found only when it's needed, by which time the redundant copy may be bad too. A scrubber reads everything periodically, verifies checksums, and repairs mismatches from redundancy while it still can. S3 says it regularly verifies data integrity with checksums and repairs lost redundancy; ZFS runs scrubs to catch "bit rot".

When the CPU is wrong

Checksums catch bits that change at rest or in transit. A CPU that computes the wrong result produces wrong data with a correct checksum, because the checksum is computed over the wrong data. Defences are expensive and partial: compute critical results twice on different machines and compare, check invariants ("a compressed file can't have size zero"), and screen hardware with test suites to retire bad cores.

Recap

Interview questions