Post 02 · DDIA Ch. 3–4

Storage & Evolution

The world's simplest database is two lines of bash. It fails immediately. Every real storage engine is a different bet on which pain you'd rather pay for — and once you've picked one, you still have to change the schema without a stop-the-world.

Bytes on disk, then bytes on the wire. How data sits, and how it survives two versions of your code running at once.

Append every write to the end of a file. That's it — that's the whole database. Writes are O(1) and stay in disk order, which is the fastest thing spinning rust can do. Reads are the problem: without an index, you scan the whole file. Every storage engine that follows is an attempt to add an index without giving up sequential writes.

Hash index: keep every key's location in a RAM map. Fastest point lookup possible; the whole index must fit in memory. B-tree: a balanced on-disk tree, updated in place — random I/O on writes, but scans and transactions are natural. LSM-tree: buffer writes in memory, flush to sorted files on disk, compact in the background — sequential writes at the cost of write amplification. Every mainstream OLTP database is one of these three.

OLTP fetches one whole record ("give me user 42's profile"). Row-oriented storage puts the whole record on one page — one seek, done. OLAP scans one column across millions of rows ("average order value last quarter"). Column-oriented storage puts each column's values contiguous on disk, which turns the scan into streaming reads and compresses beautifully because adjacent values are the same type. Wrong layout, ten to a hundred times slower.

Now you know how a byte gets to disk. Next question: what shape is that byte, and what happens when tomorrow's code deploys next to today's?

Same record, six encodings. JSON is 81 bytes and self-describing; Protobuf is 33 bytes but needs the schema to read. The size gap looks small until you multiply it by billions of messages per day and the CPU cost of parsing text. The trade is universal: text formats are readable and forgiving; binary formats are compact and fast but demand schema discipline.

A rolling deploy means old code and new code run at the same time, hitting the same database and the same message queue. Backward compatibility: new code reads old data. Forward compatibility: old code reads new data — and must ignore fields it doesn't know exist. Both directions have to hold for the full deploy window, or something writes data no one can read.

Adding an optional field with a default is always safe. Renaming a field breaks JSON but not Protobuf (the tag number stays). Removing a required field breaks everyone. The specifics vary by format; the discipline is the same: change one thing at a time, deploy readers before writers, keep old tag numbers reserved.