storage: byte documents in a per-collection slab (roadmap item 4)

Documents live as canonical BSON bytes in a segmented per-collection slab
(fixed 8 MiB segments keep capacity slack under one segment); the docs map
holds flat offsets that stay valid across segment growth, and removed
documents leave garbage bytes until compaction rewrites. The per-document
ArenaAllocator and its second full Pair-tree copy are gone.

The matcher walks the stored bytes directly, skipping by length any field
the filter does not name (a new bson byte-walker: element_key, skip_value,
read_value with borrowed leaves, get_at, and a borrowed spine parse). The
byte matcher is differential-tested against the tree matcher on a corpus
and shares its operator logic. Stored documents are never materialized on
the scan path or in aggregate $match; $group reads group keys and sums
straight off the bytes. Sort, projection, findAndModify, updates and
index entry generation use a borrowed spine into the slab (or the byte
collector, which also replaced collect_values in build_entries). The
compaction threshold now counts uncompressed data volume, since a
compressed log would otherwise never trigger.

Measured (tests/e2e/results/phase5.txt): server RSS 1979 -> 539 MB (2.4x
smaller than MongoDB; phase1 baseline 2.0 GB), range-scan 22.5 -> ~12 ms
(parity, best run faster than MongoDB), proj 4.1 -> 3.4 ms, createIndex
parity. Verified: unit suite in all three modes with zero leaks, the
crash pair, e2e6, and the stress/spill programs.
This commit is contained in:
2026-08-02 22:15:07 +03:00
parent b4585106f1
commit 570900a6ef
12 changed files with 985 additions and 253 deletions

View File

@@ -1,13 +1,12 @@
# Remaining performance work
Status: **items 1 (B+tree), 2 (ordered `_id` index) and 3 (block-framed
compressed log) are done** — verified in `tests/e2e/results/phase2.txt`
through `phase4.txt`: updateMany 17.3 → 1.6 ms, createIndex 62 → 51 ms,
`_id` sort+limit 6.2 → 2.4 ms, and `db on disk` 1025 → 97 MB (smaller
than MongoDB's own compressed files; bulk insert 816 → 722 MB/s, the
accepted compression cost). Item 4's dependent (item 1) now stands on a
tree instead of a sorted array. Items below, in dependency order.
Each is sized to be landed and verified on
Status: **items 1 (B+tree), 2 (ordered `_id` index), 3 (block-framed
compressed log) and 4 (byte storage) are done** — verified in
`tests/e2e/results/phase2.txt` through `phase5.txt`: updateMany 17.3 →
1.7 ms, createIndex 62 → 51 ms, `_id` sort+limit 6.2 → 2.4 ms, `db on
disk` 1025 → 97 MB, and `server RSS` 1979 → 539 MB with the range scan at
parity (best run faster than MongoDB). Only item 5 (decompose the global
lock) remains. Each is sized to be landed and verified on
its own; the ordering constraints between them are the load-bearing part, so
read those before picking one up.
@@ -208,7 +207,24 @@ compression for free — but it must still defer syncing and commit once.
---
## 4. Stop giving every document its own arena
## 4. Stop giving every document its own arena — DONE
Landed: documents live as canonical BSON bytes in a segmented per-collection
slab (fixed segments keep capacity slack under one segment; the docs map
holds flat offsets, stable across growth). The matcher walks the bytes
directly, skipping by length any field the filter does not name, and is
differential-tested against the tree matcher on a corpus; `$match` in
aggregate and the scan path never materialize stored documents. Sort,
projection, updates, findAndModify and index entry generation use a
borrowed spine (or the byte collector) into the slab. `bson.Document` keeps
its arena-backed tree meaning for transient docs; stored docs are
represented by their bytes.
Recorded deltas vs `tests/e2e/results/phase4.txt`: `server RSS` 1979 →
539 MB (2.4x smaller than MongoDB); `range-scan` 22.5 → ~12 ms (parity;
best run 11.2 vs 14.0); `proj` 4.1 → 3.4 ms. Verified with `zig build
test` in all three modes (zero leaks; the byte matcher differential), the
crash pair, e2e6, and the stress/spill programs.
**Why.** Two gaps at once. RSS is 2.0 GB against 1.4 GB for a 1.0 GB dataset
because each document carries an `ArenaAllocator` and a second full copy of