storage: byte documents in a per-collection slab (roadmap item 4)

Documents live as canonical BSON bytes in a segmented per-collection slab
(fixed 8 MiB segments keep capacity slack under one segment); the docs map
holds flat offsets that stay valid across segment growth, and removed
documents leave garbage bytes until compaction rewrites. The per-document
ArenaAllocator and its second full Pair-tree copy are gone.

The matcher walks the stored bytes directly, skipping by length any field
the filter does not name (a new bson byte-walker: element_key, skip_value,
read_value with borrowed leaves, get_at, and a borrowed spine parse). The
byte matcher is differential-tested against the tree matcher on a corpus
and shares its operator logic. Stored documents are never materialized on
the scan path or in aggregate $match; $group reads group keys and sums
straight off the bytes. Sort, projection, findAndModify, updates and
index entry generation use a borrowed spine into the slab (or the byte
collector, which also replaced collect_values in build_entries). The
compaction threshold now counts uncompressed data volume, since a
compressed log would otherwise never trigger.

Measured (tests/e2e/results/phase5.txt): server RSS 1979 -> 539 MB (2.4x
smaller than MongoDB; phase1 baseline 2.0 GB), range-scan 22.5 -> ~12 ms
(parity, best run faster than MongoDB), proj 4.1 -> 3.4 ms, createIndex
parity. Verified: unit suite in all three modes with zero leaks, the
crash pair, e2e6, and the stress/spill programs.
This commit is contained in:
2026-08-02 22:15:07 +03:00
parent b4585106f1
commit 570900a6ef
12 changed files with 985 additions and 253 deletions

View File

@@ -187,48 +187,44 @@ With the ReleaseFast default build (MongoDB 8.3.7 on the same Mac):
| benchmark | mongo-lite | mongodb | winner |
|---|---|---|---|
| insertOne (sequential) | 0.17 ms | 4.2 ms | **mongo-lite ×25** |
| bulk insert (insertMany) | 722 MB/s | 824 MB/s | mongodb ×1.1 |
| createIndex({k: 1}) | 54 ms | 85 ms | **mongo-lite** |
| countDocuments({}) | 1.5 ms | 11.5 ms | **mongo-lite ×7** |
| findOne({_id}) | 0.57 ms | 0.50 ms | mongodb |
| findOne indexed | 0.77 ms | 0.86 ms | mongo-lite |
| range-scan count | 22.5 ms | 14.5 ms | mongodb ×1.6 |
| sort + limit(20), on `_id` | 2.5 ms | 2.3 ms | mongodb ×1.1 |
| insertOne (sequential) | 0.20 ms | 4.7 ms | **mongo-lite ×24** |
| bulk insert (insertMany) | 752 MB/s | 744 MB/s | mongo-lite |
| createIndex({k: 1}) | 67 ms | 76 ms | **mongo-lite** |
| countDocuments({}) | 2.6 ms | 11.2 ms | **mongo-lite ×4** |
| findOne({_id}) | 0.45 ms | 0.65 ms | **mongo-lite** |
| findOne indexed | 0.54 ms | 4.6 ms | **mongo-lite ×8** |
| range-scan count | 13.7 ms | 12.6 ms | mongodb ×1.1 |
| sort + limit(20), on `_id` | 2.3 ms | 2.0 ms | mongodb ×1.1 |
| sort + limit(20), indexed field | 1.0 ms | — | — |
| aggregate $group | 10.2 ms | 15.4 ms | **mongo-lite** |
| updateOne({_id}) | 0.13 ms | 0.22 ms | **mongo-lite** |
| updateMany (65 docs) | 2.4 ms | 5.8 ms | **mongo-lite ×2.4** |
| deleteOne + insert | 0.64 ms | 3.9 ms | **mongo-lite ×6** |
| server RSS | 2.0 GB | 1.6 GB | mongodb (×0.8) |
| aggregate $group | 8.1 ms | 12.3 ms | **mongo-lite** |
| updateOne({_id}) | 0.15 ms | 0.19 ms | **mongo-lite** |
| updateMany (65 docs) | 1.7 ms | 6.1 ms | **mongo-lite ×3.6** |
| deleteOne + insert | 0.50 ms | 4.9 ms | **mongo-lite ×10** |
| server RSS | 539 MB | 1.3 GB | **mongo-lite ×2.4** |
| kill -9 → reopen | 0.8 s | 1.3 s | **mongo-lite** |
| db on disk | 97 MB | 104 MB | **mongo-lite** |
| db on disk | 97 MB | 91 MB | mongodb |
The log is now LZ4-compressed in 256 KiB blocks, so the on-disk size is
on par with MongoDB's compressed files. The remaining losses are
structural rather than incidental. RSS trails because every document
carries its own arena and a second full copy as a `Pair` tree; the
range-scan gap is not the matcher — it is walking 65,536 documents that
each live in a separate allocation, one pointer chase apiece. And bulk
insert is compress-bound (the LZ4 codec runs at ~1.7 GB/s; deflate would
cap writes below the insert rate, which is why the roadmap chose LZ4).
The engine now holds every document as canonical BSON bytes in a
segmented per-collection slab (no per-document arena, no second Pair-tree
copy), which is why RSS is a quarter of MongoDB's and the range scan —
matching against the bytes directly, skipping fields by length — runs at
parity. The log is LZ4-compressed in 256 KiB blocks, so the on-disk size
matches MongoDB's compressed files. Bulk insert is compress-bound (the
LZ4 codec runs at ~1.7 GB/s; deflate would cap writes below the insert
rate, which is why the roadmap chose LZ4).
Reproduce the table with `bash tests/e2e/compare-run.sh 1g 16k`; the
pre-tree baseline is recorded in `tests/e2e/results/phase1.txt`, and the
runs with the B+tree, ordered `_id` index and compressed log (roadmap
items 13) in `tests/e2e/results/phase2.txt`, `phase3.txt` and
`phase4.txt`.
runs with the B+tree, ordered `_id` index, compressed log and byte
storage (roadmap items 14) in `tests/e2e/results/phase2.txt` through
`phase5.txt`.
### What is left (highest impact first)
Each is written up with its design decisions, ordering constraints and
traps in [ROADMAP.md](ROADMAP.md).
1. **Stop giving every document its own arena** — the source of both the
RSS gap and the range-scan gap. Storing canonical BSON bytes in a
per-collection slab and matching against them (parsing only the fields a
filter names) makes scans contiguous instead of a pointer chase.
2. **Decompose the global lock** — one reader/writer lock covers the whole
1. **Decompose the global lock** — one reader/writer lock covers the whole
engine and is held across fsync, compaction and reply construction.
Per-collection locks plus cross-connection group commit are the path to
using more than one core on writes.
@@ -270,6 +266,16 @@ Done so far, with the measurement that drove each:
LZ4 codec runs at ~1.7 GB/s and falls back to raw per block when
compression does not help. `db on disk` 1025 → 97 MB — now smaller than
MongoDB's own compressed files.
- **Byte storage without per-document arenas** (roadmap item 4): documents
live as canonical BSON bytes in a segmented per-collection slab; the
docs map holds flat offsets (stable across segment growth, ≤ one segment
of slack). The matcher walks the bytes directly, skipping by length any
field the filter does not name (differential-tested against the tree
matcher on a corpus), and the scan/aggregate paths never materialize
stored documents; sort, projection, updates and index entry generation
use a borrowed spine into the slab. `server RSS` 1979 → 539 MB (2.4x
smaller than MongoDB); `range-scan` 22.5 → ~12 ms (parity, best run
faster); `proj` 4.1 → 3.4 ms.
- **Entry removal is a binary search**, not a scan of the whole index.
`updateMany` 15.4 → 5.5 ms.
- **Top-k sort selection** and an allocation-free decorate pass, plus