dev/MultiforaDB
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

index: ordered _id index (roadmap item 2)

Browse Source 
Give every Collection an implicit _id_ index (a normal Index with keys
[_id: 1]) so _id equality, $in, ranges and sorts stop depending on the
docs-map hash or a full scan. Kept out of the secondary indexes list, so
listIndexes/dropIndexes/createIndex and the log format are unchanged (no
index_create record, no double listing) and e2e3.js passes unmodified.

Maintained in upsert through the same reserve-then-insert protocol as
the secondaries, removed in evict_doc, and rebuilt after replay by
build_all_indexes alongside them (never maintained mid-replay, so a
failed add can't leave the index under-approximating). index.plan now
takes it as a separate argument. Its keys are canonical
(bson.encode_key gives int32 1, int64 1 and double 1.0 identical bytes),
so the serialization-guarded docs-map fast path (plan_id,
value_fast_path_safe and friends) is deleted.

Measured (tests/e2e/results/phase3.txt): sort({_id:-1}).limit(20) 6.2 ->
2.4 ms (2.3x slower than MongoDB -> parity); integer/string _id point
lookups, $in and ranges verified against the tree. Unit suite in all
three optimize modes, the crash pair, e2e3/e2e4/e2e6.
This commit is contained in:
Alexey Shakhmatov
2026-08-02 21:18:40 +03:00
parent 61fe952125
commit 58914a69c3
6 changed files with 267 additions and 245 deletions
Download Patch File Download Diff File Expand all files Collapse all files

38
ROADMAP.md
View File

@@ -1,9 +1,10 @@
# Remaining performance work
Status: **item 1 (B+tree over the encoded keys) is done** — landed and
verified in `tests/e2e/results/phase2.txt` (updateMany 17.3 → 1.8 ms,
createIndex 62 → 51 ms). Its dependents (items 2 and 4) now stand on a
tree instead of a sorted array. Five items below, in dependency order.
Status: **items 1 (B+tree over the encoded keys) and 2 (ordered `_id` index)
are done** — landed and verified in `tests/e2e/results/phase2.txt` and
`phase3.txt` (updateMany 17.3 → 1.6 ms, createIndex 62 → 51 ms, `_id`
sort+limit 6.2 → 2.4 ms). Their dependents (items 4) now stand on a tree
instead of a sorted array. Items below, in dependency order.
Each is sized to be landed and verified on
its own; the ordering constraints between them are the load-bearing part, so
read those before picking one up.
@@ -101,7 +102,26 @@ differential. Both already exist and both are mutation-checked.
---
## 2. Ordered `_id` index — **depends on 1**
## 2. Ordered `_id` index — DONE
Landed as an implicit `_id_` index on every `Collection` (a normal
`index.Index` with keys `[_id: 1]`, kept out of the secondary `indexes`
list so `listIndexes`/`dropIndexes`/`createIndex` and the log format are
unchanged — no `index_create` record, no double listing). Maintained in
`upsert` (through the same reserve-then-insert protocol as the
secondaries) and `evict_doc`; rebuilt after replay by `build_all_indexes`
alongside the secondaries. `index.plan` now takes it as a separate
argument, so `{_id: ...}` equality, `$in` and ranges use the tree (the
old serialization-guarded docs-map fast path — `plan_id`,
`value_fast_path_safe` and friends — is deleted), and `sort({_id: ...})`
becomes an index-ordered full scan with an early stop. A full `_id` scan
cannot miss a document: every doc has an `_id` and the index is not
sparse, and its keys are canonical (`bson.encode_key` gives int32 1,
int64 1 and double 1.0 identical bytes).
Recorded deltas vs `tests/e2e/results/phase2.txt`: `sort({_id:-1}).limit(20)`
6.2 → 2.4 ms (2.3x slower than MongoDB → parity). Integer/string `_id`
point lookups, `$in` and ranges no longer full-scan.
**Why.** `sort({_id: ...})` still materializes every candidate, and integer
`_id`s still fall back to a full collection scan on every `findOne`,
@@ -115,8 +135,8 @@ requires.
Then delete `value_fast_path_safe` and friends (`src/index.zig`). They exist
only because `serialize_value` gives `int32 1`, `int64 1` and `double 1.0`
different bytes despite comparing equal. `bson.encode_key` already gives them
identical bytes, so the guard is obsolete.
different bytes despite comparing equal. `bson.encode_key` already gives
them identical bytes, so the guard is obsolete.
**Why it depends on item 1.** This index updates on *every* insert. Against a
sorted array that is a tail memmove each time — roughly 51 GB of memmove over
@@ -128,7 +148,9 @@ only" caveat.
**Watch.** A real `_id_` index may start appearing in `listIndexes` and
writing an `index_create` record to the log. `e2e3.js` asserts on index
listings — check it before assuming this is invisible.
listings — check it before assuming this is invisible. (This landed without
either: the index stays out of the secondary list, so the listing, drop and
log surfaces are untouched; verified with `e2e3.js` unchanged.)
---