storage: byte documents in a per-collection slab (roadmap item 4)
Documents live as canonical BSON bytes in a segmented per-collection slab (fixed 8 MiB segments keep capacity slack under one segment); the docs map holds flat offsets that stay valid across segment growth, and removed documents leave garbage bytes until compaction rewrites. The per-document ArenaAllocator and its second full Pair-tree copy are gone. The matcher walks the stored bytes directly, skipping by length any field the filter does not name (a new bson byte-walker: element_key, skip_value, read_value with borrowed leaves, get_at, and a borrowed spine parse). The byte matcher is differential-tested against the tree matcher on a corpus and shares its operator logic. Stored documents are never materialized on the scan path or in aggregate $match; $group reads group keys and sums straight off the bytes. Sort, projection, findAndModify, updates and index entry generation use a borrowed spine into the slab (or the byte collector, which also replaced collect_values in build_entries). The compaction threshold now counts uncompressed data volume, since a compressed log would otherwise never trigger. Measured (tests/e2e/results/phase5.txt): server RSS 1979 -> 539 MB (2.4x smaller than MongoDB; phase1 baseline 2.0 GB), range-scan 22.5 -> ~12 ms (parity, best run faster than MongoDB), proj 4.1 -> 3.4 ms, createIndex parity. Verified: unit suite in all three modes with zero leaks, the crash pair, e2e6, and the stress/spill programs.
This commit is contained in:
66
README.md
66
README.md
@@ -187,48 +187,44 @@ With the ReleaseFast default build (MongoDB 8.3.7 on the same Mac):
|
||||
|
||||
| benchmark | mongo-lite | mongodb | winner |
|
||||
|---|---|---|---|
|
||||
| insertOne (sequential) | 0.17 ms | 4.2 ms | **mongo-lite ×25** |
|
||||
| bulk insert (insertMany) | 722 MB/s | 824 MB/s | mongodb ×1.1 |
|
||||
| createIndex({k: 1}) | 54 ms | 85 ms | **mongo-lite** |
|
||||
| countDocuments({}) | 1.5 ms | 11.5 ms | **mongo-lite ×7** |
|
||||
| findOne({_id}) | 0.57 ms | 0.50 ms | mongodb |
|
||||
| findOne indexed | 0.77 ms | 0.86 ms | mongo-lite |
|
||||
| range-scan count | 22.5 ms | 14.5 ms | mongodb ×1.6 |
|
||||
| sort + limit(20), on `_id` | 2.5 ms | 2.3 ms | mongodb ×1.1 |
|
||||
| insertOne (sequential) | 0.20 ms | 4.7 ms | **mongo-lite ×24** |
|
||||
| bulk insert (insertMany) | 752 MB/s | 744 MB/s | mongo-lite |
|
||||
| createIndex({k: 1}) | 67 ms | 76 ms | **mongo-lite** |
|
||||
| countDocuments({}) | 2.6 ms | 11.2 ms | **mongo-lite ×4** |
|
||||
| findOne({_id}) | 0.45 ms | 0.65 ms | **mongo-lite** |
|
||||
| findOne indexed | 0.54 ms | 4.6 ms | **mongo-lite ×8** |
|
||||
| range-scan count | 13.7 ms | 12.6 ms | mongodb ×1.1 |
|
||||
| sort + limit(20), on `_id` | 2.3 ms | 2.0 ms | mongodb ×1.1 |
|
||||
| sort + limit(20), indexed field | 1.0 ms | — | — |
|
||||
| aggregate $group | 10.2 ms | 15.4 ms | **mongo-lite** |
|
||||
| updateOne({_id}) | 0.13 ms | 0.22 ms | **mongo-lite** |
|
||||
| updateMany (65 docs) | 2.4 ms | 5.8 ms | **mongo-lite ×2.4** |
|
||||
| deleteOne + insert | 0.64 ms | 3.9 ms | **mongo-lite ×6** |
|
||||
| server RSS | 2.0 GB | 1.6 GB | mongodb (×0.8) |
|
||||
| aggregate $group | 8.1 ms | 12.3 ms | **mongo-lite** |
|
||||
| updateOne({_id}) | 0.15 ms | 0.19 ms | **mongo-lite** |
|
||||
| updateMany (65 docs) | 1.7 ms | 6.1 ms | **mongo-lite ×3.6** |
|
||||
| deleteOne + insert | 0.50 ms | 4.9 ms | **mongo-lite ×10** |
|
||||
| server RSS | 539 MB | 1.3 GB | **mongo-lite ×2.4** |
|
||||
| kill -9 → reopen | 0.8 s | 1.3 s | **mongo-lite** |
|
||||
| db on disk | 97 MB | 104 MB | **mongo-lite** |
|
||||
| db on disk | 97 MB | 91 MB | mongodb |
|
||||
|
||||
The log is now LZ4-compressed in 256 KiB blocks, so the on-disk size is
|
||||
on par with MongoDB's compressed files. The remaining losses are
|
||||
structural rather than incidental. RSS trails because every document
|
||||
carries its own arena and a second full copy as a `Pair` tree; the
|
||||
range-scan gap is not the matcher — it is walking 65,536 documents that
|
||||
each live in a separate allocation, one pointer chase apiece. And bulk
|
||||
insert is compress-bound (the LZ4 codec runs at ~1.7 GB/s; deflate would
|
||||
cap writes below the insert rate, which is why the roadmap chose LZ4).
|
||||
The engine now holds every document as canonical BSON bytes in a
|
||||
segmented per-collection slab (no per-document arena, no second Pair-tree
|
||||
copy), which is why RSS is a quarter of MongoDB's and the range scan —
|
||||
matching against the bytes directly, skipping fields by length — runs at
|
||||
parity. The log is LZ4-compressed in 256 KiB blocks, so the on-disk size
|
||||
matches MongoDB's compressed files. Bulk insert is compress-bound (the
|
||||
LZ4 codec runs at ~1.7 GB/s; deflate would cap writes below the insert
|
||||
rate, which is why the roadmap chose LZ4).
|
||||
|
||||
Reproduce the table with `bash tests/e2e/compare-run.sh 1g 16k`; the
|
||||
pre-tree baseline is recorded in `tests/e2e/results/phase1.txt`, and the
|
||||
runs with the B+tree, ordered `_id` index and compressed log (roadmap
|
||||
items 1–3) in `tests/e2e/results/phase2.txt`, `phase3.txt` and
|
||||
`phase4.txt`.
|
||||
runs with the B+tree, ordered `_id` index, compressed log and byte
|
||||
storage (roadmap items 1–4) in `tests/e2e/results/phase2.txt` through
|
||||
`phase5.txt`.
|
||||
|
||||
### What is left (highest impact first)
|
||||
|
||||
Each is written up with its design decisions, ordering constraints and
|
||||
traps in [ROADMAP.md](ROADMAP.md).
|
||||
|
||||
1. **Stop giving every document its own arena** — the source of both the
|
||||
RSS gap and the range-scan gap. Storing canonical BSON bytes in a
|
||||
per-collection slab and matching against them (parsing only the fields a
|
||||
filter names) makes scans contiguous instead of a pointer chase.
|
||||
2. **Decompose the global lock** — one reader/writer lock covers the whole
|
||||
1. **Decompose the global lock** — one reader/writer lock covers the whole
|
||||
engine and is held across fsync, compaction and reply construction.
|
||||
Per-collection locks plus cross-connection group commit are the path to
|
||||
using more than one core on writes.
|
||||
@@ -270,6 +266,16 @@ Done so far, with the measurement that drove each:
|
||||
LZ4 codec runs at ~1.7 GB/s and falls back to raw per block when
|
||||
compression does not help. `db on disk` 1025 → 97 MB — now smaller than
|
||||
MongoDB's own compressed files.
|
||||
- **Byte storage without per-document arenas** (roadmap item 4): documents
|
||||
live as canonical BSON bytes in a segmented per-collection slab; the
|
||||
docs map holds flat offsets (stable across segment growth, ≤ one segment
|
||||
of slack). The matcher walks the bytes directly, skipping by length any
|
||||
field the filter does not name (differential-tested against the tree
|
||||
matcher on a corpus), and the scan/aggregate paths never materialize
|
||||
stored documents; sort, projection, updates and index entry generation
|
||||
use a borrowed spine into the slab. `server RSS` 1979 → 539 MB (2.4x
|
||||
smaller than MongoDB); `range-scan` 22.5 → ~12 ms (parity, best run
|
||||
faster); `proj` 4.1 → 3.4 ms.
|
||||
- **Entry removal is a binary search**, not a scan of the whole index.
|
||||
`updateMany` 15.4 → 5.5 ms.
|
||||
- **Top-k sort selection** and an allocation-free decorate pass, plus
|
||||
|
||||
Reference in New Issue
Block a user