index: B+tree over the encoded keys (roadmap item 1)
Replace Index.entries (one sorted array) with a B+tree so writes into an already-built index stop being quadratic. Nodes are fixed 4 KiB slotted pages in a flat u32-addressed ArrayListUnmanaged(Node); records longer than a quarter page spill to an append-only overflow slab (BSON strings reach 16 MB). Leaves are doubly linked for ordered iteration; the flat node array stays one contiguous byte range for a later checkpoint. Insertion descends by separator and splits leaves/internals upward, promoting keys via a stable copy (a nested split can otherwise clobber the promoted-key scratch). Deletion does not rebalance: emptied leaves are unlinked and dropped from their parent, internal nodes may carry one child, and dead pages are abandoned in place (node memory peaks at the tree's peak size, exactly what the old array's capacity did). Lookups are lower-bound seeks plus leaf-chain band scans, so equal keys may span leaves freely. Bulk build (append_doc_entries + finish_bulk) sorts a staging array and packs leaves bottom-up. reserve_for now takes the built entries and reserves exact overflow bytes plus a worst-case node count, keeping insert_entries infallible after the log append. db.zig: TTL sweep now seeks the minimum-datetime encoded key and walks the contiguous datetime band, stopping at the cutoff or type change. Measured (tests/e2e/results/phase2.txt): updateMany 17.3 -> 1.8 ms (2.8x slower than MongoDB -> 3.7x faster), createIndex 62 -> 51 ms. Verified: unit suite ReleaseFast/ReleaseSafe/Debug (incl. the existing lookup_range and remove_doc differentials, plus a new incremental insert/remove differential against a brute-force model), the crash pair, e2e3/e2e4/e2e6, and dev stress tests for depth-2 splits, full drains, and spilled records through internal levels.
This commit is contained in:
56
README.md
56
README.md
@@ -185,22 +185,22 @@ With the ReleaseFast default build (MongoDB 8.3.7 on the same Mac):
|
||||
|
||||
| benchmark | mongo-lite | mongodb | winner |
|
||||
|---|---|---|---|
|
||||
| insertOne (sequential) | 0.19 ms | 5.0 ms | **mongo-lite ×26** |
|
||||
| bulk insert (insertMany) | 853 MB/s | 690 MB/s | **mongo-lite ×1.2** |
|
||||
| createIndex({k: 1}) | 62 ms | 78 ms | **mongo-lite** |
|
||||
| countDocuments({}) | 1.5 ms | 11.5 ms | **mongo-lite ×8** |
|
||||
| findOne({_id}) | 0.48 ms | 0.54 ms | mongo-lite |
|
||||
| findOne indexed | 0.64 ms | 4.3 ms | **mongo-lite ×7** |
|
||||
| range-scan count | 21 ms | 13 ms | mongodb ×1.7 |
|
||||
| sort + limit(20), on `_id` | 4.3 ms | 2.0 ms | mongodb ×2 |
|
||||
| insertOne (sequential) | 0.19 ms | 4.1 ms | **mongo-lite ×22** |
|
||||
| bulk insert (insertMany) | 810 MB/s | 714 MB/s | **mongo-lite ×1.1** |
|
||||
| createIndex({k: 1}) | 51 ms | 82 ms | **mongo-lite** |
|
||||
| countDocuments({}) | 1.5 ms | 13.8 ms | **mongo-lite ×9** |
|
||||
| findOne({_id}) | 0.57 ms | 0.67 ms | mongo-lite |
|
||||
| findOne indexed | 0.57 ms | 1.8 ms | **mongo-lite ×3** |
|
||||
| range-scan count | 20 ms | 13 ms | mongodb ×1.6 |
|
||||
| sort + limit(20), on `_id` | 6.2 ms | 2.7 ms | mongodb ×2.3 |
|
||||
| sort + limit(20), indexed field | 1.0 ms | — | — |
|
||||
| aggregate $group | 9.8 ms | 13.7 ms | **mongo-lite** |
|
||||
| updateOne({_id}) | 0.16 ms | 0.19 ms | mongo-lite |
|
||||
| updateMany (65 docs) | 5.5 ms | 6.1 ms | mongo-lite |
|
||||
| deleteOne + insert | 0.62 ms | 4.9 ms | **mongo-lite ×8** |
|
||||
| server RSS | 2.0 GB | 1.4 GB | mongodb (×0.7) |
|
||||
| aggregate $group | 11.5 ms | 15.5 ms | **mongo-lite** |
|
||||
| updateOne({_id}) | 0.17 ms | 0.19 ms | mongo-lite |
|
||||
| updateMany (65 docs) | 1.8 ms | 6.7 ms | **mongo-lite ×3.7** |
|
||||
| deleteOne + insert | 0.50 ms | 5.0 ms | **mongo-lite ×10** |
|
||||
| server RSS | 2.0 GB | 1.5 GB | mongodb (×0.7) |
|
||||
| kill -9 → reopen | 0.8 s | 1.3 s | **mongo-lite** |
|
||||
| db on disk | 1.0 GB | 93 MB | mongodb (compressed) |
|
||||
| db on disk | 1.0 GB | 96 MB | mongodb (compressed) |
|
||||
|
||||
The remaining losses are structural rather than incidental. Disk size is
|
||||
the big one: payloads are stored raw, so the log is 11x MongoDB's
|
||||
@@ -211,8 +211,10 @@ that each live in a separate allocation, one pointer chase apiece. And
|
||||
covers `_id` yet; the same sort on an indexed field streams straight out
|
||||
of the index at 1.0 ms.
|
||||
|
||||
Reproduce the table with `bash tests/e2e/compare-run.sh 1g 16k`; the run
|
||||
above is recorded in `tests/e2e/results/phase1.txt`.
|
||||
Reproduce the table with `bash tests/e2e/compare-run.sh 1g 16k`; the
|
||||
pre-tree baseline is recorded in `tests/e2e/results/phase1.txt`, the run
|
||||
above (with the B+tree, roadmap item 1) in
|
||||
`tests/e2e/results/phase2.txt`.
|
||||
|
||||
### What is left (highest impact first)
|
||||
|
||||
@@ -224,20 +226,15 @@ traps in [ROADMAP.md](ROADMAP.md).
|
||||
highly compressible workloads massively; note Zig 0.16 ships zstd
|
||||
decompression only, and deflate would cap writes below the current
|
||||
insert rate.
|
||||
2. **A B-tree over the encoded keys** — entry insert still memmoves the
|
||||
tail of a sorted array, so writing into a collection that already has an
|
||||
index is quadratic. A flat, `u32`-indexed node array would also be
|
||||
dumpable into a checkpoint, which is what makes a fast reopen possible.
|
||||
3. **An ordered `_id` index** — `sort({_id: ...})` still materializes every
|
||||
2. **An ordered `_id` index** — `sort({_id: ...})` still materializes every
|
||||
candidate, and integer `_id`s still scan. Both fall out of indexing the
|
||||
encoded `_id`. It wants the tree first: an `_id` index updates on every
|
||||
insert, and doing that against a sorted array is only cheap because
|
||||
ObjectIds append at the end.
|
||||
4. **Stop giving every document its own arena** — the source of both the
|
||||
encoded `_id`. The tree is in place, so an `_id` index update is a
|
||||
leaf insert, not a tail memmove.
|
||||
3. **Stop giving every document its own arena** — the source of both the
|
||||
RSS gap and the range-scan gap. Storing canonical BSON bytes in a
|
||||
per-collection slab and matching against them (parsing only the fields a
|
||||
filter names) makes scans contiguous instead of a pointer chase.
|
||||
5. **Decompose the global lock** — one reader/writer lock covers the whole
|
||||
4. **Decompose the global lock** — one reader/writer lock covers the whole
|
||||
engine and is held across fsync, compaction and reply construction.
|
||||
Per-collection locks plus cross-connection group commit are the path to
|
||||
using more than one core on writes.
|
||||
@@ -255,6 +252,13 @@ Done so far, with the measurement that drove each:
|
||||
array. `createIndex` over 65,536 documents 649 → 44 ms.
|
||||
- **Index entries hold encoded byte keys**, so comparing them is a memcmp
|
||||
rather than a walk over values in unrelated arenas.
|
||||
- **A B+tree over the encoded keys** (roadmap item 1): fixed 4 KiB slotted
|
||||
pages in a flat u32-addressed node array, an overflow slab for long
|
||||
records, no rebalancing on delete, and bulk bottom-up packing. Entry
|
||||
insertion and removal are a descent plus a leaf-local edit instead of a
|
||||
tail memmove, so writes into an already-built index stopped being
|
||||
quadratic. `updateMany` 17.3 → 1.8 ms (2.8x slower than MongoDB → 3.7x
|
||||
faster); `createIndex` 62 → 51 ms.
|
||||
- **Entry removal is a binary search**, not a scan of the whole index.
|
||||
`updateMany` 15.4 → 5.5 ms.
|
||||
- **Top-k sort selection** and an allocation-free decorate pass, plus
|
||||
|
||||
Reference in New Issue
Block a user