Aleksey Shakhmatov
b4585106f1
The log is now a 16-byte file header (magic, version, codec, block target) plus a sequence of blocks. Each block keeps the pre-existing record framing unchanged, so Engine.apply_record does not change; records never straddle blocks (appends accumulate in memory and the block seals at ~256 KiB). The block header's integrity hash covers the stored payload bytes exactly as they sit on disk, so the decompressor only ever sees input already proven intact. Torn tails stay distinguishable from interior corruption exactly as before: a short read, an impossible length, or a hash mismatch in the final block truncates cleanly (later appends overwrite the garbage); a hash mismatch anywhere else is error.InvalidLog. The codec is a hand-rolled LZ4 block compressor/decompressor (~1.7 GB/s measured) with a per-block codec byte falling back to raw when compression does not help; the header keeps raw legal so zstd can be swapped in later. Zig 0.16 ships zstd decompression only, and deflate would cap writes below the insert rate. Engine.compact goes through the same Log API (deferred sync, one commit) and compresses for free; sync() seals the pending block before fsyncing, so the acknowledged-write durability semantics are unchanged (an unsealed block holds only unacknowledged batch records). Measured (tests/e2e/results/phase4.txt): db on disk 1025 -> 97 MB, now smaller than MongoDB's own compressed files; bulk insert 816 -> 722 MB/s (the accepted compression cost); reopen unchanged at 0.8 s. Verified: unit suite in all three optimize modes (new LZ4 round-trip, corrupt-block, and torn-tail truncation tests), the crash pair, e2e6 (kill -9 mid-write), and two full benchmark runs.
3.3 KiB
3.3 KiB