storage: block-framed LZ4-compressed log (roadmap item 3)

The log is now a 16-byte file header (magic, version, codec, block
target) plus a sequence of blocks. Each block keeps the pre-existing
record framing unchanged, so Engine.apply_record does not change; records
never straddle blocks (appends accumulate in memory and the block seals
at ~256 KiB). The block header's integrity hash covers the stored payload
bytes exactly as they sit on disk, so the decompressor only ever sees
input already proven intact. Torn tails stay distinguishable from
interior corruption exactly as before: a short read, an impossible
length, or a hash mismatch in the final block truncates cleanly (later
appends overwrite the garbage); a hash mismatch anywhere else is
error.InvalidLog.

The codec is a hand-rolled LZ4 block compressor/decompressor (~1.7 GB/s
measured) with a per-block codec byte falling back to raw when
compression does not help; the header keeps raw legal so zstd can be
swapped in later. Zig 0.16 ships zstd decompression only, and deflate
would cap writes below the insert rate.

Engine.compact goes through the same Log API (deferred sync, one commit)
and compresses for free; sync() seals the pending block before fsyncing,
so the acknowledged-write durability semantics are unchanged (an
unsealed block holds only unacknowledged batch records).

Measured (tests/e2e/results/phase4.txt): db on disk 1025 -> 97 MB, now
smaller than MongoDB's own compressed files; bulk insert 816 -> 722 MB/s
(the accepted compression cost); reopen unchanged at 0.8 s.

Verified: unit suite in all three optimize modes (new LZ4 round-trip,
corrupt-block, and torn-tail truncation tests), the crash pair, e2e6
(kill -9 mid-write), and two full benchmark runs.
This commit is contained in:
2026-08-02 21:35:22 +03:00
parent 58914a69c3
commit b4585106f1
4 changed files with 629 additions and 117 deletions

View File

@@ -1,10 +1,12 @@
# Remaining performance work
Status: **items 1 (B+tree over the encoded keys) and 2 (ordered `_id` index)
are done** — landed and verified in `tests/e2e/results/phase2.txt` and
`phase3.txt` (updateMany 17.3 → 1.6 ms, createIndex 62 → 51 ms, `_id`
sort+limit 6.2 → 2.4 ms). Their dependents (items 4) now stand on a tree
instead of a sorted array. Items below, in dependency order.
Status: **items 1 (B+tree), 2 (ordered `_id` index) and 3 (block-framed
compressed log) are done** — verified in `tests/e2e/results/phase2.txt`
through `phase4.txt`: updateMany 17.3 → 1.6 ms, createIndex 62 → 51 ms,
`_id` sort+limit 6.2 → 2.4 ms, and `db on disk` 1025 → 97 MB (smaller
than MongoDB's own compressed files; bulk insert 816 → 722 MB/s, the
accepted compression cost). Item 4's dependent (item 1) now stands on a
tree instead of a sorted array. Items below, in dependency order.
Each is sized to be landed and verified on
its own; the ordering constraints between them are the load-bearing part, so
read those before picking one up.
@@ -154,7 +156,27 @@ log surfaces are untouched; verified with `e2e3.js` unchanged.)
---
## 3. Block-framed compressed log
## 3. Block-framed compressed log — DONE
Landed in `src/storage.zig`: a 16-byte file header (magic, version, codec,
block target) plus a sequence of 16-byte-header blocks, each holding the
pre-existing record framing unchanged (`Engine.apply_record` untouched),
with the integrity hash covering the stored payload bytes so the
decompressor only ever sees input already proven intact. ~256 KiB target;
records never straddle blocks (appends accumulate in memory and the block
seals when the next record would push it past the target). A short read,
an impossible length, or a hash mismatch in the final block truncates
cleanly; a mismatch elsewhere is `error.InvalidLog`. A hand-rolled LZ4
block codec (~1.7 GB/s measured) with a per-block codec byte falling back
to raw when compression does not help. `Engine.compact` goes through the
same `Log` API (deferred sync, one commit) and compresses for free.
Recorded deltas vs `tests/e2e/results/phase3.txt`: `db on disk` 1025 →
97 MB (now smaller than MongoDB's own 104 MB); bulk insert 816 → 722 MB/s
(the compression cost, accepted per the codec note below); reopen 0.8 s
unchanged. Verified with `zig build test` in all three modes (new LZ4
round-trip, corrupt-block and torn-tail tests), the crash pair, e2e6
(kill -9 mid-write), and two full benchmark runs.
**Why.** The largest remaining gap: 1.0 GB on disk against MongoDB's 93 MB,
because payloads are stored raw. Breaking the format is fine.