storage: block-framed LZ4-compressed log (roadmap item 3)

The log is now a 16-byte file header (magic, version, codec, block
target) plus a sequence of blocks. Each block keeps the pre-existing
record framing unchanged, so Engine.apply_record does not change; records
never straddle blocks (appends accumulate in memory and the block seals
at ~256 KiB). The block header's integrity hash covers the stored payload
bytes exactly as they sit on disk, so the decompressor only ever sees
input already proven intact. Torn tails stay distinguishable from
interior corruption exactly as before: a short read, an impossible
length, or a hash mismatch in the final block truncates cleanly (later
appends overwrite the garbage); a hash mismatch anywhere else is
error.InvalidLog.

The codec is a hand-rolled LZ4 block compressor/decompressor (~1.7 GB/s
measured) with a per-block codec byte falling back to raw when
compression does not help; the header keeps raw legal so zstd can be
swapped in later. Zig 0.16 ships zstd decompression only, and deflate
would cap writes below the insert rate.

Engine.compact goes through the same Log API (deferred sync, one commit)
and compresses for free; sync() seals the pending block before fsyncing,
so the acknowledged-write durability semantics are unchanged (an
unsealed block holds only unacknowledged batch records).

Measured (tests/e2e/results/phase4.txt): db on disk 1025 -> 97 MB, now
smaller than MongoDB's own compressed files; bulk insert 816 -> 722 MB/s
(the accepted compression cost); reopen unchanged at 0.8 s.

Verified: unit suite in all three optimize modes (new LZ4 round-trip,
corrupt-block, and torn-tail truncation tests), the crash pair, e2e6
(kill -9 mid-write), and two full benchmark runs.

README.md

@@ -51,12 +51,14 @@ mongosh --port 27017
   over-approximates is merely slow, never wrong.
 - **Update operators**: `$set` `$unset` `$inc` `$push` (`$each`) `$pull`
   `$rename`, with dot-path creation (including array indices).
-- **Storage**: append-only record log (CRC32-checked, `fsync` per write,
-  torn-tail tolerant) with in-memory indexes rebuilt on open and automatic
-  compaction (rewrite + atomic rename when the log grows past
-  `--compact-threshold`, default 16 MB). Killed mid-write (`kill -9`), the
-  database recovers all committed writes; the log and compaction both work
-  with relative or absolute `--db` paths. Records up to the announced 16 MB
+- **Storage**: append-only record log, LZ4-compressed in 256 KiB blocks
+  (XxHash3-checked, `fsync` per write, torn-tail tolerant: a crash
+  mid-append truncates cleanly, interior corruption is rejected) with
+  in-memory indexes rebuilt on open and automatic compaction (rewrite +
+  atomic rename when the log grows past `--compact-threshold`, default
+  16 MB). Killed mid-write (`kill -9`), the database recovers all
+  committed writes; the log and compaction both work with relative or
+  absolute `--db` paths. Records up to the announced 16 MB
   `maxBsonObjectSize` replay correctly.
 - **Concurrency**: a writer-preferring read/write lock splits command
   execution — reads (`find`, `count`, `aggregate`, `list*`) run concurrently
@@ -185,49 +187,48 @@ With the ReleaseFast default build (MongoDB 8.3.7 on the same Mac):
 
 | benchmark | mongo-lite | mongodb | winner |
 |---|---|---|---|
-| insertOne (sequential) | 0.19 ms | 4.1 ms | **mongo-lite ×22** |
-| bulk insert (insertMany) | 810 MB/s | 714 MB/s | **mongo-lite ×1.1** |
-| createIndex({k: 1}) | 51 ms | 82 ms | **mongo-lite** |
-| countDocuments({}) | 1.5 ms | 13.8 ms | **mongo-lite ×9** |
-| findOne({_id}) | 0.57 ms | 0.67 ms | mongo-lite |
-| findOne indexed | 0.58 ms | 1.5 ms | **mongo-lite ×2.6** |
-| range-scan count | 22 ms | 13 ms | mongodb ×1.7 |
-| sort + limit(20), on `_id` | 2.4 ms | 2.2 ms | mongodb ×1.1 |
+| insertOne (sequential) | 0.17 ms | 4.2 ms | **mongo-lite ×25** |
+| bulk insert (insertMany) | 722 MB/s | 824 MB/s | mongodb ×1.1 |
+| createIndex({k: 1}) | 54 ms | 85 ms | **mongo-lite** |
+| countDocuments({}) | 1.5 ms | 11.5 ms | **mongo-lite ×7** |
+| findOne({_id}) | 0.57 ms | 0.50 ms | mongodb |
+| findOne indexed | 0.77 ms | 0.86 ms | mongo-lite |
+| range-scan count | 22.5 ms | 14.5 ms | mongodb ×1.6 |
+| sort + limit(20), on `_id` | 2.5 ms | 2.3 ms | mongodb ×1.1 |
 | sort + limit(20), indexed field | 1.0 ms | — | — |
-| aggregate $group | 11.5 ms | 15.5 ms | **mongo-lite** |
-| updateOne({_id}) | 0.17 ms | 0.19 ms | mongo-lite |
-| updateMany (65 docs) | 1.6 ms | 6.4 ms | **mongo-lite ×4** |
-| deleteOne + insert | 0.62 ms | 4.8 ms | **mongo-lite ×8** |
-| server RSS | 2.0 GB | 1.5 GB | mongodb (×0.7) |
+| aggregate $group | 10.2 ms | 15.4 ms | **mongo-lite** |
+| updateOne({_id}) | 0.13 ms | 0.22 ms | **mongo-lite** |
+| updateMany (65 docs) | 2.4 ms | 5.8 ms | **mongo-lite ×2.4** |
+| deleteOne + insert | 0.64 ms | 3.9 ms | **mongo-lite ×6** |
+| server RSS | 2.0 GB | 1.6 GB | mongodb (×0.8) |
 | kill -9 → reopen | 0.8 s | 1.3 s | **mongo-lite** |
-| db on disk | 1.0 GB | 96 MB | mongodb (compressed) |
+| db on disk | 97 MB | 104 MB | **mongo-lite** |
 
-The remaining losses are structural rather than incidental. Disk size is
-the big one: payloads are stored raw, so the log is 11x MongoDB's
-compressed files. RSS trails because every document carries its own arena.
-The range-scan gap is not the matcher — it is walking 65,536 documents
-that each live in a separate allocation, one pointer chase apiece.
+The log is now LZ4-compressed in 256 KiB blocks, so the on-disk size is
+on par with MongoDB's compressed files. The remaining losses are
+structural rather than incidental. RSS trails because every document
+carries its own arena and a second full copy as a `Pair` tree; the
+range-scan gap is not the matcher — it is walking 65,536 documents that
+each live in a separate allocation, one pointer chase apiece. And bulk
+insert is compress-bound (the LZ4 codec runs at ~1.7 GB/s; deflate would
+cap writes below the insert rate, which is why the roadmap chose LZ4).
 
 Reproduce the table with `bash tests/e2e/compare-run.sh 1g 16k`; the
 pre-tree baseline is recorded in `tests/e2e/results/phase1.txt`, and the
-runs with the B+tree and ordered `_id` index (roadmap items 1 and 2) in
-`tests/e2e/results/phase2.txt` and `tests/e2e/results/phase3.txt`.
+runs with the B+tree, ordered `_id` index and compressed log (roadmap
+items 1–3) in `tests/e2e/results/phase2.txt`, `phase3.txt` and
+`phase4.txt`.
 
 ### What is left (highest impact first)
 
 Each is written up with its design decisions, ordering constraints and
 traps in [ROADMAP.md](ROADMAP.md).
 
-1. **Compress the log** — the largest remaining gap (×11). Payloads are
-   stored raw. A block-framed format with an LZ4 block codec would shrink
-   highly compressible workloads massively; note Zig 0.16 ships zstd
-   decompression only, and deflate would cap writes below the current
-   insert rate.
-2. **Stop giving every document its own arena** — the source of both the
+1. **Stop giving every document its own arena** — the source of both the
    RSS gap and the range-scan gap. Storing canonical BSON bytes in a
    per-collection slab and matching against them (parsing only the fields a
    filter names) makes scans contiguous instead of a pointer chase.
-3. **Decompose the global lock** — one reader/writer lock covers the whole
+2. **Decompose the global lock** — one reader/writer lock covers the whole
    engine and is held across fsync, compaction and reply construction.
    Per-collection locks plus cross-connection group commit are the path to
    using more than one core on writes.
@@ -260,6 +261,15 @@ Done so far, with the measurement that drove each:
   `_id` point lookups, `$in` and ranges hit the tree instead of a full
   scan. `sort({_id: ...})` is now an index-ordered scan with an early stop:
   `sort+limit(20)` 6.2 → 2.4 ms (parity with MongoDB).
+- **A block-framed, LZ4-compressed log** (roadmap item 3): a file header
+  plus ~256 KiB blocks, each holding the existing record framing with the
+  integrity hash covering the stored bytes (so the decompressor only ever
+  sees input already proven intact). Records never straddle blocks; a
+  short read, impossible length or hash mismatch in the final block is a
+  torn tail (truncate cleanly), anywhere else is corruption. The hand-rolled
+  LZ4 codec runs at ~1.7 GB/s and falls back to raw per block when
+  compression does not help. `db on disk` 1025 → 97 MB — now smaller than
+  MongoDB's own compressed files.
 - **Entry removal is a binary search**, not a scan of the whole index.
   `updateMany` 15.4 → 5.5 ms.
 - **Top-k sort selection** and an allocation-free decorate pass, plus