sehaj/learn
System design/04 Databases and data modeling

04.10 · concept · 12 min

NoSQL families

Key-value, document, wide-column, graph, search and analytics stores: the shape of data each holds, the queries each is fast at, and what each gives up compared with a relational database.

"Not only SQL"

Relational databases are general-purpose: any table can be joined to any other, queried by any column, inside transactions. That generality has costs at the extremes, mostly around scaling writes across many machines and data that doesn't fit tables well.

"NoSQL" names a family of databases that each give up some of that generality to be very good at one shape of data and one set of queries. The shared idea: design the data around the queries you'll run, because the database won't be able to answer arbitrary new ones efficiently.

The families

FamilyStoresFast atExamples
Key-valueA value (often opaque) per keyGet and put by exact key, at huge scaleRedis, DynamoDB, Memcached
DocumentJSON-like documents, nestedFetching and updating a whole document by ID; queries on its fieldsMongoDB, Couchbase, Firestore
Wide-columnRows grouped by a partition key, sorted within itVery high write rates; reading a range within one partitionCassandra, ScyllaDB, Bigtable, HBase
GraphNodes and the edges between themFollowing relationships many hops deepNeo4j, Neptune
SearchInverted indexes of text and fieldsFull-text search, ranking, filtering by many fieldsElasticsearch, OpenSearch
Analytics (columnar)Columns stored separately, compressedScanning and aggregating billions of rows on a few columnsClickHouse, BigQuery, Redshift

A note on the last two: search and columnar stores are rarely the source of truth. They're usually fed from the main database, for queries it can't do well.

What a partition key changes

Most NoSQL stores that scale out (key-value and wide-column especially) ask you to choose a partition key for each table. The store hashes it to decide which machine holds the data (05.01 Sharding: range, hash, directory). Then:

  • Reads and writes by partition key go straight to one machine: fast at any size.
  • Queries within one partition (say, "messages in channel 7, newest first") are fast if the data is sorted by a second key inside it.
  • Queries across partitions ("all messages containing 'hello'") are slow or impossible. There's no planner to rescue you (04.03 Reading a query plan).

So you model one table per query if you have to, copying data into each shape (04.12 Denormalisation), instead of one normalised model that joins at read time.

For Snip

Snip dataShape and queriesA good fit
Links (code → URL)Get by exact key, a lotRelational with a cache today; key-value at enormous scale
Users, plans, billingRelations, constraints, transactionsRelational
Raw click eventsBillions of appends, scanned for chartsColumnar analytics, or wide-column
Link titles and URLs, searchableFull textSearch index, fed from the main database

What you give up

Each family drops some of what relational databases give, and it's worth naming:

  • Joins. Usually gone or limited. Data you'd join is copied together instead.
  • Ad hoc queries. New questions may need new tables or a full scan.
  • Transactions across items. Often limited to one key or one partition, or slower when offered.
  • Consistency. Many default to eventually consistent reads (01.03 Availability, durability, consistency, latency, cost), with stronger options at a cost.
  • Schema enforcement. "Schemaless" means the schema lives in your code, and old documents in old shapes live forever unless you migrate them.

Seen in production

Amazon

2022

DynamoDB

The paper says that during the 66-hour Amazon Prime Day event in 2021, Amazon systems made trillions of API calls to DynamoDB, peaking at 89.2 million requests per second, while seeing high availability and single-digit millisecond performance. The abstract opens with its central promise: consistent performance at any scale.

What we foundA key-value store built for predictable latency at any scale handled tens of millions of requests a second at peak.

Amazon DynamoDB: A Scalable, Predictably Performant, and Fully Managed NoSQL Database Service (USENIX ATC 2022)usenix.org ↗
Predict first

A chat app stores messages in a wide-column store, partitioned by channel_id and sorted by time. A new feature needs 'all messages by user 42 across every channel'. What's the usual answer?

Commit to an answer before you read on.

Recap

Interview questions

MidfreqMust-knowConcept

When would you choose a NoSQL database over a relational one?

Report a mistakeEdit this page