sehaj/learn
System design/15 Classic designs

15.04 · design · 25 min

Design: a key-value store

A Dynamo-style distributed key-value store: a hash ring with virtual nodes, N replicas with tunable R and W, sloppy quorums and hinted handoff to stay writable through failures (tested), conflict handling, an LSM engine on each node, gossip and anti-entropy.

The prompt

"Design a key-value store" tests whether you can put the distributed-systems building blocks together into one system, and explain what each costs. The classic reference is Amazon's Dynamo.

Design it yourself firstseniorconsistent hashingquorumsreplicationconflict resolution

Build a store with two operations, get(key) and put(key, value), for values up to 1 MB. It holds 100 TB of data, serves 1 million operations a second with p99 under 10 ms, and must always accept writes, even when some machines or a network link fail. Different applications want different trade-offs between consistency and availability.

25:00

1. Requirements

Functional: get and put by key; deletes; per-application settings for consistency.

Non-functional:

  • Always writable: a shopping cart that rejects "add to cart" loses sales.
  • p99 under 10 ms; 1 million operations a second.
  • Grows by adding machines, with no downtime and little data moved.
  • No single point of failure, and no single coordinator.

Out of scope: range queries, secondary indexes, transactions across keys.

2. Estimates

  • Storage: 100 TB × 3 replicas = 300 TB. At about 4 TB usable per node, kept under 70% full for compaction and repair, that's about 100 nodes.
  • Load: 1 million operations a second over 100 nodes is 10,000 per node; with three replicas involved in each, about 30,000 replica operations per node. Each node is an ordinary server with SSDs.

3. API

text
get(key)                    → { values: [value, …], context }    one value normally, several on conflict
put(key, value, context?)   → ok                                 context: the version the writer read

The context carries version information (a version vector, 08.05 Clocks and ordering), so the store can tell an update that saw the previous value from one that didn't.

4. High-level design

A Dynamo-style store
if one is downgossipClientCoordinatorany nodeReplica 1LSM engineReplica 2LSM engineReplica 3LSM engineStand-inholds hints
There's no leader: any node coordinates any request, and every key's replicas are found the same way, by walking the ring.

5 steps. Use Next and Back to follow the request.

Text version
  1. The client sends a put to any node, which acts as the coordinator. Every node knows the ring, learned by gossip.
  2. The coordinator hashes the key and walks the ring clockwise to its preference list: the first N = 3 distinct physical nodes.
  3. It sends the write to all three and answers the client once W = 2 have acknowledged. Each replica appends to its write-ahead log and memtable.
  4. If a replica is down, the next healthy node on the ring takes the write with a hint naming the real owner, and hands it back when the owner returns.
  5. Nodes gossip membership and health, and compare Merkle trees in the background to repair replicas that drifted.
  • Client → Coordinator
  • Coordinator → Replica 1
  • Coordinator → Replica 2
  • Coordinator → Replica 3
  • Coordinator → Stand-in: if one is down
  • Replica 1 ↔ Replica 2: gossip

5. Deep dives

Deep dive

Spreading keys across 100 nodes

Badhash(key) mod number of nodes

Adding one node changes the result for almost every key: a 100 TB reshuffle.

GoodConsistent hashing

Nodes sit on a ring; a key belongs to the next node clockwise. Adding a node moves only the keys between it and its neighbour (05.03 Consistent hashing). But with one position per node, load is uneven, and a new node takes all its keys from one neighbour.

GreatConsistent hashing with virtual nodes

Each physical node takes many positions on the ring, so load evens out, and a joining node takes small pieces from every other node. Bigger machines get more positions.

Deep dive

Staying writable when replicas fail

BadOne primary per key

Simple and consistent, but when the primary is unreachable that key can't be written until a new one is chosen.

GoodStrict quorums: W of the N owners

With N = 3 and W = 2, one owner can be down. With R = 2 as well, R + W > N, so every read overlaps the latest write (08.04 Replication and quorums). But with two owners down, writes fail.

GreatSloppy quorums with hinted handoff

Count acknowledgements from the first W healthy nodes on the ring, owners or not. A stand-in stores the write with a hint and returns it to the owner when it recovers. Writes keep working through failures; the price is that a read can briefly miss a write held by a stand-in.

The core of it, with five nodes, 64 virtual nodes each, N = 3 and W = R = 2:

ts
type Versioned = { value: string; version: number };
const hash32 = (s: string) => createHash("md5").update(s).digest().readUInt32BE(0);

export class KvCluster {
  private ring: [number, string][] = []; // (position, physical node), sorted
  data = new Map<string, Map<string, Versioned>>();
  hints = new Map<string, { owner: string; key: string; v: Versioned }[]>();
  down = new Set<string>();

  constructor(nodes: string[], readonly n = 3, readonly w = 2, readonly r = 2, vnodes = 64) {
    for (const node of nodes) {
      this.data.set(node, new Map());
      for (let i = 0; i < vnodes; i++) this.ring.push([hash32(`${node}#${i}`), node]);
    }
    this.ring.sort((a, b) => a[0] - b[0]);
  }

  /** Walk clockwise from the key, collecting distinct physical nodes. */
  walk(key: string): string[] {
    const h = hash32(key);
    let i = this.ring.findIndex(([p]) => p >= h);
    if (i < 0) i = 0;
    const out: string[] = [];
    for (let k = 0; out.length < this.data.size && k < this.ring.length; k++) {
      const node = this.ring[(i + k) % this.ring.length][1];
      if (!out.includes(node)) out.push(node);
    }
    return out;
  }

  put(key: string, value: string, version: number) {
    const all = this.walk(key);
    const spares = all.slice(this.n).filter((s) => !this.down.has(s));
    let acks = 0;
    for (const owner of all.slice(0, this.n)) {
      const v = { value, version };
      if (!this.down.has(owner)) {
        this.store(owner, key, v);
        acks++;
      } else if (spares.length) {
        const holder = spares.shift()!; // sloppy quorum: a healthy stand-in keeps it, with a hint
        this.store(holder, key, v);
        this.hints.set(holder, [...(this.hints.get(holder) ?? []), { owner, key, v }]);
        acks++;
      }
    }
    if (acks < this.w) throw new Error(`only ${acks} of ${this.w} writes acknowledged`);
  }

  get(key: string) {
    const replies = this.walk(key).filter((node) => !this.down.has(node)).slice(0, this.n)
      .map((node) => this.data.get(node)!.get(key));
    if (replies.length < this.r) throw new Error("not enough replicas");
    return replies.filter(Boolean).sort((a, b) => b!.version - a!.version)[0];
  }

  /** When a node comes back, every stand-in hands its hints over. */
  recover(node: string) {
    this.down.delete(node);
    for (const [holder, list] of this.hints) {
      this.hints.set(holder, list.filter((h) => (h.owner === node ? (this.store(node, h.key, h.v), false) : true)));
    }
  }

  private store(node: string, key: string, v: Versioned) {
    const cur = this.data.get(node)!.get(key);
    if (!cur || cur.version < v.version) this.data.get(node)!.set(key, v);
  }
}

Tested: user:42's preference list was three distinct nodes, B, E and A. With B down, the write of version 2 succeeded, D held it with a hint, and a read returned version 2 while B itself still had version 1. When B recovered, the hint was handed over and B had version 2. With all three owners down, the write was still accepted by the two remaining nodes: availability, at the cost that the owners know nothing about it until they're back. Adding a sixth node moved 1,416 of 10,000 keys' primary owner (one sixth would be about 1,667), every one of them to the new node.

Deep dive

Two writes to the same key at once

BadLast write wins by each server's clock

Clocks drift (05.06 Clocks, skew and ordering IDs): a write stamped by a fast clock silently beats a later one. Data is lost without anyone noticing.

GoodLast write wins, chosen deliberately

For data where losing a concurrent update is acceptable (a profile photo, a session), it's simple and needs no help from the application. Say so explicitly.

GreatVersion vectors, with merging

Each value carries a version vector; the store can tell "B saw A" from "A and B happened concurrently" (08.05 Clocks and ordering). Concurrent versions are both kept and returned to the client, which merges them (a cart's union) and writes the result back. Data types that merge automatically (CRDTs) remove the client step.

Each node's storage engine. Writes dominate and must be fast, so each node runs an LSM engine: append to the write-ahead log, insert into the memtable, flush sorted files, compact in the background (09.03 The LSM write and read path). A Bloom filter per file keeps reads for missing keys cheap (05.08 Bloom filters). Deletes are tombstones, kept long enough to reach every replica, or a replica that missed the delete would bring the value back.

Failure detection and repair. Nodes learn membership and health by gossip (08.10 Gossip and membership). Replicas drift when hints are lost; read repair fixes stale replicas noticed during reads, and anti-entropy compares Merkle trees between replicas in the background and sends only the ranges that differ.

6. Wrap-up

The knobs are the point. N, R and W per table let a cart choose (3, 2, 2) and a cache choose (3, 1, 1). What breaks next is usually operational: repair traffic after a long outage, compaction falling behind, and hot keys, which no ring spreads (05.02 Hot partitions and celebrity keys).

What's expected at each level

Junior

gets a working design

  • Partitions with consistent hashing and replicates each key

Mid-level

explains the trade-offs

  • Explains R + W > N and picks values
  • Adds virtual nodes and explains why
  • Chooses an LSM engine for a write-heavy store

Senior

drives it and goes deep

  • Uses sloppy quorums and hinted handoff, and states what they cost
  • Handles conflicts with version vectors, or chooses last-write-wins knowingly
  • Covers gossip, read repair, anti-entropy and tombstones

Seen in production

Amazon

2007

Dynamo

Dynamo lets each instance tune N, R and W, noting that R + W > N gives a quorum-like system, and the paper gives (3, 2, 2) as the common configuration used by several instances. To stay writable during failures it uses a sloppy quorum with hinted handoff, and it tracks causality between versions of an object with vector clocks, returning conflicting versions to the application to reconcile.

What we foundTunable N, R and W, with (3, 2, 2) as the common choice, plus sloppy quorums and hinted handoff for availability.

Dynamo: Amazon's Highly Available Key-value Store (SOSP 2007)allthingsdistributed.com ↗

Apache Cassandra

Docs

Tunable consistency

Cassandra describes its consistency levels as a version of Dynamo's R + W > N mechanism: ONE needs a single replica, QUORUM a majority of replicas, and ALL every replica. Replicas converge through hinted handoff, read repair, and anti-entropy repair, in which replicas compute Merkle trees over their data.

What we foundDynamo's R + W > N, exposed as per-request consistency levels.

Dynamo (Apache Cassandra documentation)cassandra.apache.org ↗
Predict first

A store runs N = 3, W = 1, R = 1 to be as fast as possible. A client writes a key and immediately reads it back from a different coordinator. What can happen?

Commit to an answer before you read on.

Recap

Interview questions

SeniorfreqMust-knowDesign

Design a distributed key-value store.

Report a mistakeEdit this page