How object storage works

The problem: bytes that don't belong in a database

Snip now produces three kinds of file:

• QR code images for every paid link, about 5 KB each, millions of them, fetched by browsers.
• CSV exports of an owner's click history, from kilobytes to several gigabytes.
• Archived click history, compressed monthly files kept for years, rarely read.

Postgres could hold them as large values, but it would bloat backups, replication and the buffer cache with data that doesn't need transactions or indexes. A shared file system on the app servers doesn't scale past a few machines and is one more thing to keep available. This is what object storage is for: Amazon S3, Google Cloud Storage, Azure Blob Storage, or self-hosted MinIO and Ceph.

The model

• Buckets hold objects. Each object has a key (qr/x7Kp.png, exports/4821/2026-11.csv), its bytes, and a little metadata (content type, custom tags).
• The namespace is flat. The slashes in keys look like folders, but there are no directories; listing by prefix just filters keys.
• Objects are written and read whole. You can read a byte range, but you can't change byte 1,000 of an object; you replace the object. That immutability is what makes the system simple to replicate, cache and scale.
• Operations are few: PUT, GET, DELETE, LIST by prefix, plus multipart upload for big objects (upload parts in parallel, retry failed parts, then combine) and pre-signed URLs, which let a browser upload or download directly without passing through your servers (13.05 Signed URLs and access control).

On S3 today, reads after a successful write, overwrite or delete always return the latest version, including LIST (08.02 Consistency models).

How it's built underneath

The systems differ, but the shape is common:

1. A front end authenticates the request and finds where the object lives.
2. A metadata service maps each key to its location: which storage nodes, which files, which offsets. It's a huge, partitioned key-value store in its own right (05.03 Consistent hashing).
3. Storage nodes hold the bytes, usually split into pieces with redundancy across machines, racks and data centres, either by replication or erasure coding (09.05 Replication vs erasure coding).

Small objects: don't use a file per object

The naive design stores each object as a file on a storage node's file system. With billions of small files, the problem isn't data; it's file-system metadata. Finding a file's blocks means reading its inode and directory entries, often from disk, before reading the data. Facebook's photo store hit exactly this: reading metadata became the throughput bottleneck.

Their answer, Haystack, packs many photos ("needles") into a few huge append-only files, and keeps a compact in-memory index from photo ID to file offset: the log-and-index design from 09.01 A storage engine is a log and an index, so a read needs at most one disk operation. Modern object stores pack small objects into large files in a similar spirit.

Spreading the load

A hard disk does only about a hundred random operations a second, so the key to performance at scale is spreading each customer's data over very many drives, so no single disk gets hot. Andy Warfield, describing S3, calls this "heat management": objects are deliberately placed on different sets of drives, so each customer's requests are spread over a very large number of disks, and S3 has tens of thousands of customers whose buckets span millions of drives.

Storage classes and lifecycle

Object stores price by how often you read. Snip's archives move automatically, by lifecycle rules on their key prefix, from the standard class to an infrequent-access class after 30 days and to an archive class after a year: cheaper to keep, slower and costlier to read. Exports are deleted after 30 days.

Recap

Interview questions