storage: block-framed LZ4-compressed log (roadmap item 3)
The log is now a 16-byte file header (magic, version, codec, block target) plus a sequence of blocks. Each block keeps the pre-existing record framing unchanged, so Engine.apply_record does not change; records never straddle blocks (appends accumulate in memory and the block seals at ~256 KiB). The block header's integrity hash covers the stored payload bytes exactly as they sit on disk, so the decompressor only ever sees input already proven intact. Torn tails stay distinguishable from interior corruption exactly as before: a short read, an impossible length, or a hash mismatch in the final block truncates cleanly (later appends overwrite the garbage); a hash mismatch anywhere else is error.InvalidLog. The codec is a hand-rolled LZ4 block compressor/decompressor (~1.7 GB/s measured) with a per-block codec byte falling back to raw when compression does not help; the header keeps raw legal so zstd can be swapped in later. Zig 0.16 ships zstd decompression only, and deflate would cap writes below the insert rate. Engine.compact goes through the same Log API (deferred sync, one commit) and compresses for free; sync() seals the pending block before fsyncing, so the acknowledged-write durability semantics are unchanged (an unsealed block holds only unacknowledged batch records). Measured (tests/e2e/results/phase4.txt): db on disk 1025 -> 97 MB, now smaller than MongoDB's own compressed files; bulk insert 816 -> 722 MB/s (the accepted compression cost); reopen unchanged at 0.8 s. Verified: unit suite in all three optimize modes (new LZ4 round-trip, corrupt-block, and torn-tail truncation tests), the crash pair, e2e6 (kill -9 mid-write), and two full benchmark runs.
This commit is contained in:
34
ROADMAP.md
34
ROADMAP.md
@@ -1,10 +1,12 @@
|
||||
# Remaining performance work
|
||||
|
||||
Status: **items 1 (B+tree over the encoded keys) and 2 (ordered `_id` index)
|
||||
are done** — landed and verified in `tests/e2e/results/phase2.txt` and
|
||||
`phase3.txt` (updateMany 17.3 → 1.6 ms, createIndex 62 → 51 ms, `_id`
|
||||
sort+limit 6.2 → 2.4 ms). Their dependents (items 4) now stand on a tree
|
||||
instead of a sorted array. Items below, in dependency order.
|
||||
Status: **items 1 (B+tree), 2 (ordered `_id` index) and 3 (block-framed
|
||||
compressed log) are done** — verified in `tests/e2e/results/phase2.txt`
|
||||
through `phase4.txt`: updateMany 17.3 → 1.6 ms, createIndex 62 → 51 ms,
|
||||
`_id` sort+limit 6.2 → 2.4 ms, and `db on disk` 1025 → 97 MB (smaller
|
||||
than MongoDB's own compressed files; bulk insert 816 → 722 MB/s, the
|
||||
accepted compression cost). Item 4's dependent (item 1) now stands on a
|
||||
tree instead of a sorted array. Items below, in dependency order.
|
||||
Each is sized to be landed and verified on
|
||||
its own; the ordering constraints between them are the load-bearing part, so
|
||||
read those before picking one up.
|
||||
@@ -154,7 +156,27 @@ log surfaces are untouched; verified with `e2e3.js` unchanged.)
|
||||
|
||||
---
|
||||
|
||||
## 3. Block-framed compressed log
|
||||
## 3. Block-framed compressed log — DONE
|
||||
|
||||
Landed in `src/storage.zig`: a 16-byte file header (magic, version, codec,
|
||||
block target) plus a sequence of 16-byte-header blocks, each holding the
|
||||
pre-existing record framing unchanged (`Engine.apply_record` untouched),
|
||||
with the integrity hash covering the stored payload bytes so the
|
||||
decompressor only ever sees input already proven intact. ~256 KiB target;
|
||||
records never straddle blocks (appends accumulate in memory and the block
|
||||
seals when the next record would push it past the target). A short read,
|
||||
an impossible length, or a hash mismatch in the final block truncates
|
||||
cleanly; a mismatch elsewhere is `error.InvalidLog`. A hand-rolled LZ4
|
||||
block codec (~1.7 GB/s measured) with a per-block codec byte falling back
|
||||
to raw when compression does not help. `Engine.compact` goes through the
|
||||
same `Log` API (deferred sync, one commit) and compresses for free.
|
||||
|
||||
Recorded deltas vs `tests/e2e/results/phase3.txt`: `db on disk` 1025 →
|
||||
97 MB (now smaller than MongoDB's own 104 MB); bulk insert 816 → 722 MB/s
|
||||
(the compression cost, accepted per the codec note below); reopen 0.8 s
|
||||
unchanged. Verified with `zig build test` in all three modes (new LZ4
|
||||
round-trip, corrupt-block and torn-tail tests), the crash pair, e2e6
|
||||
(kill -9 mid-write), and two full benchmark runs.
|
||||
|
||||
**Why.** The largest remaining gap: 1.0 GB on disk against MongoDB's 93 MB,
|
||||
because payloads are stored raw. Breaking the format is fine.
|
||||
|
||||
Reference in New Issue
Block a user