index: ordered _id index (roadmap item 2)

Give every Collection an implicit _id_ index (a normal Index with keys
[_id: 1]) so _id equality, $in, ranges and sorts stop depending on the
docs-map hash or a full scan. Kept out of the secondary indexes list, so
listIndexes/dropIndexes/createIndex and the log format are unchanged (no
index_create record, no double listing) and e2e3.js passes unmodified.

Maintained in upsert through the same reserve-then-insert protocol as
the secondaries, removed in evict_doc, and rebuilt after replay by
build_all_indexes alongside them (never maintained mid-replay, so a
failed add can't leave the index under-approximating). index.plan now
takes it as a separate argument. Its keys are canonical
(bson.encode_key gives int32 1, int64 1 and double 1.0 identical bytes),
so the serialization-guarded docs-map fast path (plan_id,
value_fast_path_safe and friends) is deleted.

Measured (tests/e2e/results/phase3.txt): sort({_id:-1}).limit(20) 6.2 ->
2.4 ms (2.3x slower than MongoDB -> parity); integer/string _id point
lookups, $in and ranges verified against the tree. Unit suite in all
three optimize modes, the crash pair, e2e3/e2e4/e2e6.
This commit is contained in:
2026-08-02 21:18:40 +03:00
parent 61fe952125
commit 58914a69c3
6 changed files with 267 additions and 245 deletions

View File

@@ -190,14 +190,14 @@ With the ReleaseFast default build (MongoDB 8.3.7 on the same Mac):
| createIndex({k: 1}) | 51 ms | 82 ms | **mongo-lite** |
| countDocuments({}) | 1.5 ms | 13.8 ms | **mongo-lite ×9** |
| findOne({_id}) | 0.57 ms | 0.67 ms | mongo-lite |
| findOne indexed | 0.57 ms | 1.8 ms | **mongo-lite ×3** |
| range-scan count | 20 ms | 13 ms | mongodb ×1.6 |
| sort + limit(20), on `_id` | 6.2 ms | 2.7 ms | mongodb ×2.3 |
| findOne indexed | 0.58 ms | 1.5 ms | **mongo-lite ×2.6** |
| range-scan count | 22 ms | 13 ms | mongodb ×1.7 |
| sort + limit(20), on `_id` | 2.4 ms | 2.2 ms | mongodb ×1.1 |
| sort + limit(20), indexed field | 1.0 ms | — | — |
| aggregate $group | 11.5 ms | 15.5 ms | **mongo-lite** |
| updateOne({_id}) | 0.17 ms | 0.19 ms | mongo-lite |
| updateMany (65 docs) | 1.8 ms | 6.7 ms | **mongo-lite ×3.7** |
| deleteOne + insert | 0.50 ms | 5.0 ms | **mongo-lite ×10** |
| updateMany (65 docs) | 1.6 ms | 6.4 ms | **mongo-lite ×4** |
| deleteOne + insert | 0.62 ms | 4.8 ms | **mongo-lite ×8** |
| server RSS | 2.0 GB | 1.5 GB | mongodb (×0.7) |
| kill -9 → reopen | 0.8 s | 1.3 s | **mongo-lite** |
| db on disk | 1.0 GB | 96 MB | mongodb (compressed) |
@@ -206,15 +206,12 @@ The remaining losses are structural rather than incidental. Disk size is
the big one: payloads are stored raw, so the log is 11x MongoDB's
compressed files. RSS trails because every document carries its own arena.
The range-scan gap is not the matcher — it is walking 65,536 documents
that each live in a separate allocation, one pointer chase apiece. And
`sort` on `_id` still materializes candidates because nothing ordered
covers `_id` yet; the same sort on an indexed field streams straight out
of the index at 1.0 ms.
that each live in a separate allocation, one pointer chase apiece.
Reproduce the table with `bash tests/e2e/compare-run.sh 1g 16k`; the
pre-tree baseline is recorded in `tests/e2e/results/phase1.txt`, the run
above (with the B+tree, roadmap item 1) in
`tests/e2e/results/phase2.txt`.
pre-tree baseline is recorded in `tests/e2e/results/phase1.txt`, and the
runs with the B+tree and ordered `_id` index (roadmap items 1 and 2) in
`tests/e2e/results/phase2.txt` and `tests/e2e/results/phase3.txt`.
### What is left (highest impact first)
@@ -226,15 +223,11 @@ traps in [ROADMAP.md](ROADMAP.md).
highly compressible workloads massively; note Zig 0.16 ships zstd
decompression only, and deflate would cap writes below the current
insert rate.
2. **An ordered `_id` index**`sort({_id: ...})` still materializes every
candidate, and integer `_id`s still scan. Both fall out of indexing the
encoded `_id`. The tree is in place, so an `_id` index update is a
leaf insert, not a tail memmove.
3. **Stop giving every document its own arena** — the source of both the
2. **Stop giving every document its own arena** — the source of both the
RSS gap and the range-scan gap. Storing canonical BSON bytes in a
per-collection slab and matching against them (parsing only the fields a
filter names) makes scans contiguous instead of a pointer chase.
4. **Decompose the global lock** — one reader/writer lock covers the whole
3. **Decompose the global lock** — one reader/writer lock covers the whole
engine and is held across fsync, compaction and reply construction.
Per-collection locks plus cross-connection group commit are the path to
using more than one core on writes.
@@ -257,8 +250,16 @@ Done so far, with the measurement that drove each:
records, no rebalancing on delete, and bulk bottom-up packing. Entry
insertion and removal are a descent plus a leaf-local edit instead of a
tail memmove, so writes into an already-built index stopped being
quadratic. `updateMany` 17.3 → 1.8 ms (2.8x slower than MongoDB → 3.7x
quadratic. `updateMany` 17.3 → 1.6 ms (2.8x slower than MongoDB → 4x
faster); `createIndex` 62 → 51 ms.
- **An ordered `_id` index** (roadmap item 2): every collection carries an
implicit `_id_` index (kept out of the secondary list, so the listing,
drop and log-format surfaces are unchanged; rebuilt after replay like
the secondaries). Its encoded keys are canonical, so the old
serialization-guarded docs-map fast path is gone and integer/string
`_id` point lookups, `$in` and ranges hit the tree instead of a full
scan. `sort({_id: ...})` is now an index-ordered scan with an early stop:
`sort+limit(20)` 6.2 → 2.4 ms (parity with MongoDB).
- **Entry removal is a binary search**, not a scan of the whole index.
`updateMany` 15.4 → 5.5 ms.
- **Top-k sort selection** and an allocation-free decorate pass, plus