storage: byte documents in a per-collection slab (roadmap item 4)
Documents live as canonical BSON bytes in a segmented per-collection slab (fixed 8 MiB segments keep capacity slack under one segment); the docs map holds flat offsets that stay valid across segment growth, and removed documents leave garbage bytes until compaction rewrites. The per-document ArenaAllocator and its second full Pair-tree copy are gone. The matcher walks the stored bytes directly, skipping by length any field the filter does not name (a new bson byte-walker: element_key, skip_value, read_value with borrowed leaves, get_at, and a borrowed spine parse). The byte matcher is differential-tested against the tree matcher on a corpus and shares its operator logic. Stored documents are never materialized on the scan path or in aggregate $match; $group reads group keys and sums straight off the bytes. Sort, projection, findAndModify, updates and index entry generation use a borrowed spine into the slab (or the byte collector, which also replaced collect_values in build_entries). The compaction threshold now counts uncompressed data volume, since a compressed log would otherwise never trigger. Measured (tests/e2e/results/phase5.txt): server RSS 1979 -> 539 MB (2.4x smaller than MongoDB; phase1 baseline 2.0 GB), range-scan 22.5 -> ~12 ms (parity, best run faster than MongoDB), proj 4.1 -> 3.4 ms, createIndex parity. Verified: unit suite in all three modes with zero leaks, the crash pair, e2e6, and the stress/spill programs.
This commit is contained in:
34
ROADMAP.md
34
ROADMAP.md
@@ -1,13 +1,12 @@
|
||||
# Remaining performance work
|
||||
|
||||
Status: **items 1 (B+tree), 2 (ordered `_id` index) and 3 (block-framed
|
||||
compressed log) are done** — verified in `tests/e2e/results/phase2.txt`
|
||||
through `phase4.txt`: updateMany 17.3 → 1.6 ms, createIndex 62 → 51 ms,
|
||||
`_id` sort+limit 6.2 → 2.4 ms, and `db on disk` 1025 → 97 MB (smaller
|
||||
than MongoDB's own compressed files; bulk insert 816 → 722 MB/s, the
|
||||
accepted compression cost). Item 4's dependent (item 1) now stands on a
|
||||
tree instead of a sorted array. Items below, in dependency order.
|
||||
Each is sized to be landed and verified on
|
||||
Status: **items 1 (B+tree), 2 (ordered `_id` index), 3 (block-framed
|
||||
compressed log) and 4 (byte storage) are done** — verified in
|
||||
`tests/e2e/results/phase2.txt` through `phase5.txt`: updateMany 17.3 →
|
||||
1.7 ms, createIndex 62 → 51 ms, `_id` sort+limit 6.2 → 2.4 ms, `db on
|
||||
disk` 1025 → 97 MB, and `server RSS` 1979 → 539 MB with the range scan at
|
||||
parity (best run faster than MongoDB). Only item 5 (decompose the global
|
||||
lock) remains. Each is sized to be landed and verified on
|
||||
its own; the ordering constraints between them are the load-bearing part, so
|
||||
read those before picking one up.
|
||||
|
||||
@@ -208,7 +207,24 @@ compression for free — but it must still defer syncing and commit once.
|
||||
|
||||
---
|
||||
|
||||
## 4. Stop giving every document its own arena
|
||||
## 4. Stop giving every document its own arena — DONE
|
||||
|
||||
Landed: documents live as canonical BSON bytes in a segmented per-collection
|
||||
slab (fixed segments keep capacity slack under one segment; the docs map
|
||||
holds flat offsets, stable across growth). The matcher walks the bytes
|
||||
directly, skipping by length any field the filter does not name, and is
|
||||
differential-tested against the tree matcher on a corpus; `$match` in
|
||||
aggregate and the scan path never materialize stored documents. Sort,
|
||||
projection, updates, findAndModify and index entry generation use a
|
||||
borrowed spine (or the byte collector) into the slab. `bson.Document` keeps
|
||||
its arena-backed tree meaning for transient docs; stored docs are
|
||||
represented by their bytes.
|
||||
|
||||
Recorded deltas vs `tests/e2e/results/phase4.txt`: `server RSS` 1979 →
|
||||
539 MB (2.4x smaller than MongoDB); `range-scan` 22.5 → ~12 ms (parity;
|
||||
best run 11.2 vs 14.0); `proj` 4.1 → 3.4 ms. Verified with `zig build
|
||||
test` in all three modes (zero leaks; the byte matcher differential), the
|
||||
crash pair, e2e6, and the stress/spill programs.
|
||||
|
||||
**Why.** Two gaps at once. RSS is 2.0 GB against 1.4 GB for a 1.0 GB dataset
|
||||
because each document carries an `ArenaAllocator` and a second full copy of
|
||||
|
||||
Reference in New Issue
Block a user