dev/MultiforaDB
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

storage: block-framed LZ4-compressed log (roadmap item 3)

Browse Source 
The log is now a 16-byte file header (magic, version, codec, block
target) plus a sequence of blocks. Each block keeps the pre-existing
record framing unchanged, so Engine.apply_record does not change; records
never straddle blocks (appends accumulate in memory and the block seals
at ~256 KiB). The block header's integrity hash covers the stored payload
bytes exactly as they sit on disk, so the decompressor only ever sees
input already proven intact. Torn tails stay distinguishable from
interior corruption exactly as before: a short read, an impossible
length, or a hash mismatch in the final block truncates cleanly (later
appends overwrite the garbage); a hash mismatch anywhere else is
error.InvalidLog.

The codec is a hand-rolled LZ4 block compressor/decompressor (~1.7 GB/s
measured) with a per-block codec byte falling back to raw when
compression does not help; the header keeps raw legal so zstd can be
swapped in later. Zig 0.16 ships zstd decompression only, and deflate
would cap writes below the insert rate.

Engine.compact goes through the same Log API (deferred sync, one commit)
and compresses for free; sync() seals the pending block before fsyncing,
so the acknowledged-write durability semantics are unchanged (an
unsealed block holds only unacknowledged batch records).

Measured (tests/e2e/results/phase4.txt): db on disk 1025 -> 97 MB, now
smaller than MongoDB's own compressed files; bulk insert 816 -> 722 MB/s
(the accepted compression cost); reopen unchanged at 0.8 s.

Verified: unit suite in all three optimize modes (new LZ4 round-trip,
corrupt-block, and torn-tail truncation tests), the crash pair, e2e6
(kill -9 mid-write), and two full benchmark runs.
This commit is contained in:
Alexey Shakhmatov
2026-08-02 21:35:22 +03:00
parent 58914a69c3
commit b4585106f1
4 changed files with 629 additions and 117 deletions
Download Patch File Download Diff File Expand all files Collapse all files

45
tests/e2e/results/phase4.txt Normal file
View File

@@ -0,0 +1,45 @@
# Phase 4 gate — mongo-lite vs MongoDB 8.3.7, 1g dataset / ~16k docs
# Ratio < 1.0 = mongo-lite faster. Reproduce: bash tests/e2e/compare-run.sh 1g 16k
# This run includes roadmap items 1 (B+tree), 2 (ordered _id index) and
# 3 (block-framed LZ4-compressed log).
# Compare: tests/e2e/results/phase1.txt (pre-tree baseline).
benchmark mongo-lite mongodb ratio
insertOne (sequential) ×200 0.17 ms 4.2 ms 0.0x
bulk insert throughput 722.1 MB/s 824.1 MB/s 0.9x
docs loaded 65,536 65,536 1.0x
createIndex({k: 1}) 53.9 ms 84.8 ms 0.6x
countDocuments({}) 1.5 ms 11.5 ms 0.1x
findOne({_id: <ObjectId>}) 0.57 ms 0.50 ms 1.1x
findOne({k: 500}) (indexed) 0.77 ms 0.86 ms 0.9x
find({p: {$gte,$lt}}).count() (scan) 22.5 ms 14.5 ms 1.6x
find({}).sort({_id:-1}).limit(20) 2.5 ms 2.3 ms 1.1x
find({}, {proj}).limit(1000) 4.1 ms 4.4 ms 0.9x
aggregate $group by k 10.2 ms 15.4 ms 0.7x
updateOne({_id}) ×50 0.13 ms 0.22 ms 0.6x
updateMany({k: 7}, {$inc}) 2.4 ms 5.8 ms 0.4x
deleteOne({_id}) + insertOne 0.64 ms 3.9 ms 0.2x
node client RSS 160 MB 154 MB 1.0x
server RSS 1979 MB 1561 MB
kill -9 reopen 0.8s 1.3s
db on disk 97MB 104MB
# Item 3 (block-framed LZ4-compressed log) deltas vs phase3:
# db on disk 1025 -> 97 MB (11x smaller; now smaller than MongoDB's
# own 104 MB). The log is a file header plus ~256 KiB
# blocks; each block holds the existing record framing
# (Engine.apply_record unchanged) with the integrity hash
# covering the stored bytes, so the decompressor only sees
# input already proven intact. Torn tails (short read,
# impossible length, or hash mismatch in the final block)
# truncate cleanly; hash mismatches elsewhere are
# InvalidLog. Raw blocks remain legal (per-block codec
# byte) when compression does not help.
# bulk insert 816 -> 722 MB/s (0.9x of MongoDB): the compression cost,
# accepted per the roadmap (LZ4 chosen precisely because
# deflate would cap writes below the insert rate).
# reopen 0.8 s (unchanged; decompression of 97 MB is fast).
#
# Remaining gaps and where they are addressed:
# server RSS 1.3x -> Phase 4 (per-document arena -> byte storage)
# range-scan 1.6x -> Phase 4 (contiguous byte storage, not the matcher)