dev/MultiforaDB
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

index: ordered _id index (roadmap item 2)

Browse Source 
Give every Collection an implicit _id_ index (a normal Index with keys
[_id: 1]) so _id equality, $in, ranges and sorts stop depending on the
docs-map hash or a full scan. Kept out of the secondary indexes list, so
listIndexes/dropIndexes/createIndex and the log format are unchanged (no
index_create record, no double listing) and e2e3.js passes unmodified.

Maintained in upsert through the same reserve-then-insert protocol as
the secondaries, removed in evict_doc, and rebuilt after replay by
build_all_indexes alongside them (never maintained mid-replay, so a
failed add can't leave the index under-approximating). index.plan now
takes it as a separate argument. Its keys are canonical
(bson.encode_key gives int32 1, int64 1 and double 1.0 identical bytes),
so the serialization-guarded docs-map fast path (plan_id,
value_fast_path_safe and friends) is deleted.

Measured (tests/e2e/results/phase3.txt): sort({_id:-1}).limit(20) 6.2 ->
2.4 ms (2.3x slower than MongoDB -> parity); integer/string _id point
lookups, $in and ranges verified against the tree. Unit suite in all
three optimize modes, the crash pair, e2e3/e2e4/e2e6.
This commit is contained in:
Alexey Shakhmatov
2026-08-02 21:18:40 +03:00
parent 61fe952125
commit 58914a69c3
6 changed files with 267 additions and 245 deletions
Download Patch File Download Diff File Expand all files Collapse all files

39
README.md
View File

@@ -190,14 +190,14 @@ With the ReleaseFast default build (MongoDB 8.3.7 on the same Mac):
| createIndex({k: 1}) | 51 ms | 82 ms | **mongo-lite** | | createIndex({k: 1}) | 51 ms | 82 ms | **mongo-lite** |
| countDocuments({}) | 1.5 ms | 13.8 ms | **mongo-lite ×9** | | countDocuments({}) | 1.5 ms | 13.8 ms | **mongo-lite ×9** |
| findOne({_id}) | 0.57 ms | 0.67 ms | mongo-lite | | findOne({_id}) | 0.57 ms | 0.67 ms | mongo-lite |
| findOne indexed | 0.57 ms | 1.8 ms | **mongo-lite ×3** | | findOne indexed | 0.58 ms | 1.5 ms | **mongo-lite ×2.6** |
| range-scan count | 20 ms | 13 ms | mongodb ×1.6 | | range-scan count | 22 ms | 13 ms | mongodb ×1.7 |
| sort + limit(20), on `_id` | 6.2 ms | 2.7 ms | mongodb ×2.3 | | sort + limit(20), on `_id` | 2.4 ms | 2.2 ms | mongodb ×1.1 |
| sort + limit(20), indexed field | 1.0 ms | — | — | | sort + limit(20), indexed field | 1.0 ms | — | — |
| aggregate $group | 11.5 ms | 15.5 ms | **mongo-lite** | | aggregate $group | 11.5 ms | 15.5 ms | **mongo-lite** |
| updateOne({_id}) | 0.17 ms | 0.19 ms | mongo-lite | | updateOne({_id}) | 0.17 ms | 0.19 ms | mongo-lite |
| updateMany (65 docs) | 1.8 ms | 6.7 ms | **mongo-lite ×3.7** | | updateMany (65 docs) | 1.6 ms | 6.4 ms | **mongo-lite ×4** |
| deleteOne + insert | 0.50 ms | 5.0 ms | **mongo-lite ×10** | | deleteOne + insert | 0.62 ms | 4.8 ms | **mongo-lite ×8** |
| server RSS | 2.0 GB | 1.5 GB | mongodb (×0.7) | | server RSS | 2.0 GB | 1.5 GB | mongodb (×0.7) |
| kill -9 → reopen | 0.8 s | 1.3 s | **mongo-lite** | | kill -9 → reopen | 0.8 s | 1.3 s | **mongo-lite** |
| db on disk | 1.0 GB | 96 MB | mongodb (compressed) | | db on disk | 1.0 GB | 96 MB | mongodb (compressed) |
@@ -206,15 +206,12 @@ The remaining losses are structural rather than incidental. Disk size is
the big one: payloads are stored raw, so the log is 11x MongoDB's the big one: payloads are stored raw, so the log is 11x MongoDB's
compressed files. RSS trails because every document carries its own arena. compressed files. RSS trails because every document carries its own arena.
The range-scan gap is not the matcher — it is walking 65,536 documents The range-scan gap is not the matcher — it is walking 65,536 documents
that each live in a separate allocation, one pointer chase apiece. And that each live in a separate allocation, one pointer chase apiece.
`sort` on `_id` still materializes candidates because nothing ordered
covers `_id` yet; the same sort on an indexed field streams straight out
of the index at 1.0 ms.
Reproduce the table with `bash tests/e2e/compare-run.sh 1g 16k`; the Reproduce the table with `bash tests/e2e/compare-run.sh 1g 16k`; the
pre-tree baseline is recorded in `tests/e2e/results/phase1.txt`, the run pre-tree baseline is recorded in `tests/e2e/results/phase1.txt`, and the
above (with the B+tree, roadmap item 1) in runs with the B+tree and ordered `_id` index (roadmap items 1 and 2) in
`tests/e2e/results/phase2.txt`. `tests/e2e/results/phase2.txt` and `tests/e2e/results/phase3.txt`.
### What is left (highest impact first) ### What is left (highest impact first)
@@ -226,15 +223,11 @@ traps in [ROADMAP.md](ROADMAP.md).
highly compressible workloads massively; note Zig 0.16 ships zstd highly compressible workloads massively; note Zig 0.16 ships zstd
decompression only, and deflate would cap writes below the current decompression only, and deflate would cap writes below the current
insert rate. insert rate.
2. **An ordered `_id` index** — `sort({_id: ...})` still materializes every 2. **Stop giving every document its own arena** — the source of both the
candidate, and integer `_id`s still scan. Both fall out of indexing the
encoded `_id`. The tree is in place, so an `_id` index update is a
leaf insert, not a tail memmove.
3. **Stop giving every document its own arena** — the source of both the
RSS gap and the range-scan gap. Storing canonical BSON bytes in a RSS gap and the range-scan gap. Storing canonical BSON bytes in a
per-collection slab and matching against them (parsing only the fields a per-collection slab and matching against them (parsing only the fields a
filter names) makes scans contiguous instead of a pointer chase. filter names) makes scans contiguous instead of a pointer chase.
4. **Decompose the global lock** — one reader/writer lock covers the whole 3. **Decompose the global lock** — one reader/writer lock covers the whole
engine and is held across fsync, compaction and reply construction. engine and is held across fsync, compaction and reply construction.
Per-collection locks plus cross-connection group commit are the path to Per-collection locks plus cross-connection group commit are the path to
using more than one core on writes. using more than one core on writes.
@@ -257,8 +250,16 @@ Done so far, with the measurement that drove each:
records, no rebalancing on delete, and bulk bottom-up packing. Entry records, no rebalancing on delete, and bulk bottom-up packing. Entry
insertion and removal are a descent plus a leaf-local edit instead of a insertion and removal are a descent plus a leaf-local edit instead of a
tail memmove, so writes into an already-built index stopped being tail memmove, so writes into an already-built index stopped being
quadratic. `updateMany` 17.3 → 1.8 ms (2.8x slower than MongoDB → 3.7x quadratic. `updateMany` 17.3 → 1.6 ms (2.8x slower than MongoDB → 4x
faster); `createIndex` 62 → 51 ms. faster); `createIndex` 62 → 51 ms.
- **An ordered `_id` index** (roadmap item 2): every collection carries an
implicit `_id_` index (kept out of the secondary list, so the listing,
drop and log-format surfaces are unchanged; rebuilt after replay like
the secondaries). Its encoded keys are canonical, so the old
serialization-guarded docs-map fast path is gone and integer/string
`_id` point lookups, `$in` and ranges hit the tree instead of a full
scan. `sort({_id: ...})` is now an index-ordered scan with an early stop:
`sort+limit(20)` 6.2 → 2.4 ms (parity with MongoDB).
- **Entry removal is a binary search**, not a scan of the whole index. - **Entry removal is a binary search**, not a scan of the whole index.
`updateMany` 15.4 → 5.5 ms. `updateMany` 15.4 → 5.5 ms.
- **Top-k sort selection** and an allocation-free decorate pass, plus - **Top-k sort selection** and an allocation-free decorate pass, plus

38
ROADMAP.md
View File

@@ -1,9 +1,10 @@
# Remaining performance work # Remaining performance work
Status: **item 1 (B+tree over the encoded keys) is done** — landed and Status: **items 1 (B+tree over the encoded keys) and 2 (ordered `_id` index)
verified in `tests/e2e/results/phase2.txt` (updateMany 17.3 → 1.8 ms, are done** — landed and verified in `tests/e2e/results/phase2.txt` and
createIndex 62 → 51 ms). Its dependents (items 2 and 4) now stand on a `phase3.txt` (updateMany 17.3 → 1.6 ms, createIndex 62 → 51 ms, `_id`
tree instead of a sorted array. Five items below, in dependency order. sort+limit 6.2 → 2.4 ms). Their dependents (items 4) now stand on a tree
instead of a sorted array. Items below, in dependency order.
Each is sized to be landed and verified on Each is sized to be landed and verified on
its own; the ordering constraints between them are the load-bearing part, so its own; the ordering constraints between them are the load-bearing part, so
read those before picking one up. read those before picking one up.
@@ -101,7 +102,26 @@ differential. Both already exist and both are mutation-checked.
--- ---
## 2. Ordered `_id` index — **depends on 1** ## 2. Ordered `_id` index — DONE
Landed as an implicit `_id_` index on every `Collection` (a normal
`index.Index` with keys `[_id: 1]`, kept out of the secondary `indexes`
list so `listIndexes`/`dropIndexes`/`createIndex` and the log format are
unchanged — no `index_create` record, no double listing). Maintained in
`upsert` (through the same reserve-then-insert protocol as the
secondaries) and `evict_doc`; rebuilt after replay by `build_all_indexes`
alongside the secondaries. `index.plan` now takes it as a separate
argument, so `{_id: ...}` equality, `$in` and ranges use the tree (the
old serialization-guarded docs-map fast path — `plan_id`,
`value_fast_path_safe` and friends — is deleted), and `sort({_id: ...})`
becomes an index-ordered full scan with an early stop. A full `_id` scan
cannot miss a document: every doc has an `_id` and the index is not
sparse, and its keys are canonical (`bson.encode_key` gives int32 1,
int64 1 and double 1.0 identical bytes).
Recorded deltas vs `tests/e2e/results/phase2.txt`: `sort({_id:-1}).limit(20)`
6.2 → 2.4 ms (2.3x slower than MongoDB → parity). Integer/string `_id`
point lookups, `$in` and ranges no longer full-scan.
**Why.** `sort({_id: ...})` still materializes every candidate, and integer **Why.** `sort({_id: ...})` still materializes every candidate, and integer
`_id`s still fall back to a full collection scan on every `findOne`, `_id`s still fall back to a full collection scan on every `findOne`,
@@ -115,8 +135,8 @@ requires.
Then delete `value_fast_path_safe` and friends (`src/index.zig`). They exist Then delete `value_fast_path_safe` and friends (`src/index.zig`). They exist
only because `serialize_value` gives `int32 1`, `int64 1` and `double 1.0` only because `serialize_value` gives `int32 1`, `int64 1` and `double 1.0`
different bytes despite comparing equal. `bson.encode_key` already gives them different bytes despite comparing equal. `bson.encode_key` already gives
identical bytes, so the guard is obsolete. them identical bytes, so the guard is obsolete.
**Why it depends on item 1.** This index updates on *every* insert. Against a **Why it depends on item 1.** This index updates on *every* insert. Against a
sorted array that is a tail memmove each time — roughly 51 GB of memmove over sorted array that is a tail memmove each time — roughly 51 GB of memmove over
@@ -128,7 +148,9 @@ only" caveat.
**Watch.** A real `_id_` index may start appearing in `listIndexes` and **Watch.** A real `_id_` index may start appearing in `listIndexes` and
writing an `index_create` record to the log. `e2e3.js` asserts on index writing an `index_create` record to the log. `e2e3.js` asserts on index
listings — check it before assuming this is invisible. listings — check it before assuming this is invisible. (This landed without
either: the index stays out of the secondary list, so the listing, drop and
log surfaces are untouched; verified with `e2e3.js` unchanged.)
--- ---

29
src/commands.zig
View File

@@ -644,31 +644,10 @@ fn scan_sorted(
const filter_doc = bson.Document{ .arena = undefined, .pairs = filter }; const filter_doc = bson.Document{ .arena = undefined, .pairs = filter };
var n: usize = 0; var n: usize = 0;
// _id_ fast path: the docs map is the _id index. Skipped when the // Index plan (the implicit _id_ index first, then the secondaries):
// queried value's compare-equivalence class is serialization-ambiguous // candidates in index order, re-filtered. The returned ids alias the
// (see index.plan_id). // docs map keys, valid under the read lock.
if (try index.plan_id(ctx.gpa, filter)) |id_plan| { if (try index.plan(ctx.gpa, &coll.id_index, coll.indexes.items, filter, sort)) |p| {
var plan = id_plan;
defer plan.deinit(ctx.gpa);
// One scratch key, rebuilt per value: a key is never needed past its
// own lookup.
var key: std.ArrayListUnmanaged(u8) = .empty;
defer key.deinit(ctx.gpa);
for (plan.values) |v| {
key.clearRetainingCapacity();
try bson.write_serialized_value(v, ctx.gpa, &key);
const doc = coll.docs.get(key.items) orelse continue;
if (!try query.matches(ctx.gpa, &filter_doc, doc)) continue;
if (out) |list| try list.append(ctx.gpa, doc);
n += 1;
if (lim != 0 and n >= lim) break;
}
return n;
}
// Secondary-index plan: candidates in index order, re-filtered. The
// returned ids alias the docs map keys, valid under the read lock.
if (try index.plan(ctx.gpa, coll.indexes.items, filter, sort)) |p| {
var plan = p; var plan = p;
defer plan.deinit(ctx.gpa); defer plan.deinit(ctx.gpa);
var ids: std.ArrayListUnmanaged([]const u8) = .empty; var ids: std.ArrayListUnmanaged([]const u8) = .empty;

75
src/db.zig
View File

@@ -12,10 +12,23 @@ const index = @import("index.zig");
pub const Collection = struct { pub const Collection = struct {
docs: std.StringHashMapUnmanaged(*bson.Document), docs: std.StringHashMapUnmanaged(*bson.Document),
/// Secondary indexes (persisted through the log).
indexes: std.ArrayListUnmanaged(index.Index), indexes: std.ArrayListUnmanaged(index.Index),
/// The implicit _id_ index: every document has an _id and it is not
/// sparse, so entry count equals document count and a full scan of it
/// cannot miss a document — which is what the sort planner's full-scan
/// plan relies on. Kept out of `indexes` so the listing/drop commands
/// and the log format are unchanged (it is rebuilt on open like
/// everything else). `bson.encode_key` keys are canonical, so it also
/// replaces the old serialization-guarded docs-map fast path for
/// integer/string/etc. _id lookups.
id_index: index.Index,
fn init() Collection { fn init(gpa: std.mem.Allocator) !Collection {
return .{ .docs = .empty, .indexes = .empty }; var self: Collection = .{ .docs = .empty, .indexes = .empty, .id_index = undefined };
const keys = [_]index.IndexKey{.{ .path = "_id", .descending = false }};
self.id_index = try index.Index.init(gpa, "_id_", &keys, false, false, null);
return self;
} }
/// The secondary index with this name, or null. The single by-name /// The secondary index with this name, or null. The single by-name
@@ -108,6 +121,7 @@ pub const Engine = struct {
// Dropping a collection turns all of its records into garbage. // Dropping a collection turns all of its records into garbage.
self.live_docs -= coll.docs.count(); self.live_docs -= coll.docs.count();
self.dead_docs += coll.docs.count(); self.dead_docs += coll.docs.count();
coll.id_index.deinit(self.gpa);
for (coll.indexes.items) |*ix| ix.deinit(self.gpa); for (coll.indexes.items) |*ix| ix.deinit(self.gpa);
coll.indexes.deinit(self.gpa); coll.indexes.deinit(self.gpa);
var doc_it = coll.docs.iterator(); var doc_it = coll.docs.iterator();
@@ -139,6 +153,7 @@ pub const Engine = struct {
/// regenerating them from it, which is far cheaper than scanning. /// regenerating them from it, which is far cheaper than scanning.
fn evict_doc(self: *Engine, coll: *Collection, id_key: []const u8) void { fn evict_doc(self: *Engine, coll: *Collection, id_key: []const u8) void {
const old = coll.docs.fetchRemove(id_key) orelse return; const old = coll.docs.fetchRemove(id_key) orelse return;
coll.id_index.remove_doc(self.gpa, old.value, old.key);
for (coll.indexes.items) |*ix| ix.remove_doc(self.gpa, old.value, old.key); for (coll.indexes.items) |*ix| ix.remove_doc(self.gpa, old.value, old.key);
old.value.*.deinit(); old.value.*.deinit();
self.gpa.destroy(old.value); self.gpa.destroy(old.value);
@@ -241,6 +256,15 @@ pub const Engine = struct {
return err; return err;
}; };
} }
{
// The implicit _id_ index, through the same protocol: reserved
// before the log append, inserted infallibly after it.
var built = try coll.id_index.build_entries(self.gpa, owned, id_key);
built_list.append(self.gpa, .{ .built = built, .ix = &coll.id_index }) catch |err| {
built.deinit(self.gpa);
return err;
};
}
// 2. The _id check, mirroring the pre-index behavior. // 2. The _id check, mirroring the pre-index behavior.
if (mode == .insert and coll.docs.contains(id_key)) return error.DuplicateKey; if (mode == .insert and coll.docs.contains(id_key)) return error.DuplicateKey;
@@ -494,7 +518,9 @@ pub const Engine = struct {
if (db.collections.getPtr(coll_name)) |coll| return coll; if (db.collections.getPtr(coll_name)) |coll| return coll;
const coll_key = try self.gpa.dupe(u8, coll_name); const coll_key = try self.gpa.dupe(u8, coll_name);
errdefer self.gpa.free(coll_key); errdefer self.gpa.free(coll_key);
try db.collections.put(self.gpa, coll_key, Collection.init()); var new_coll = try Collection.init(self.gpa);
errdefer new_coll.id_index.deinit(self.gpa);
try db.collections.put(self.gpa, coll_key, new_coll);
return db.collections.getPtr(coll_name) orelse unreachable; return db.collections.getPtr(coll_name) orelse unreachable;
} }
@@ -624,26 +650,37 @@ pub const Engine = struct {
var coll_it = db_entry.value_ptr.collections.iterator(); var coll_it = db_entry.value_ptr.collections.iterator();
while (coll_it.next()) |coll_entry| { while (coll_it.next()) |coll_entry| {
for (coll_entry.value_ptr.indexes.items) |*ix| { for (coll_entry.value_ptr.indexes.items) |*ix| {
if (ix.count() > 0) continue; // defensive try self.rebuild_index(coll_entry.value_ptr, ix);
var doc_it = coll_entry.value_ptr.docs.iterator();
while (doc_it.next()) |doc_entry| {
ix.append_doc_entries(self.gpa, doc_entry.value_ptr.*, doc_entry.key_ptr.*) catch |err| switch (err) {
error.ParallelArrays => {
std.debug.print("mongo-lite: WARNING: index '{s}' cannot index an existing document; entry skipped\n", .{ix.name});
continue;
},
else => return err,
};
}
// Tolerated, not enforced: the database must always open.
if (try ix.finish_bulk(self.gpa, false)) {
std.debug.print("mongo-lite: WARNING: unique index '{s}' has duplicate keys in existing data; duplicates not enforced for existing documents\n", .{ix.name});
}
} }
try self.rebuild_index(coll_entry.value_ptr, &coll_entry.value_ptr.id_index);
} }
} }
} }
/// Rebuild one index from the live documents. Runs after replay, so it
/// is order-independent; indexes already holding entries (maintained
/// live) are skipped defensively. A duplicate under a unique index logs
/// a loud warning and keeps the index (still correct as a candidate
/// generator; future writes are still enforced) — the database always
/// opens, leaving dropIndexes as an in-band recovery path.
fn rebuild_index(self: *Engine, coll: *Collection, ix: *index.Index) !void {
if (ix.count() > 0) return; // defensive
var doc_it = coll.docs.iterator();
while (doc_it.next()) |doc_entry| {
ix.append_doc_entries(self.gpa, doc_entry.value_ptr.*, doc_entry.key_ptr.*) catch |err| switch (err) {
error.ParallelArrays => {
std.debug.print("mongo-lite: WARNING: index '{s}' cannot index an existing document; entry skipped\n", .{ix.name});
continue;
},
else => return err,
};
}
// Tolerated, not enforced: the database must always open.
if (try ix.finish_bulk(self.gpa, false)) {
std.debug.print("mongo-lite: WARNING: unique index '{s}' has duplicate keys in existing data; duplicates not enforced for existing documents\n", .{ix.name});
}
}
/// Register an (empty) index from a persisted spec document. A repeated /// Register an (empty) index from a persisted spec document. A repeated
/// create record for the same name is an idempotent no-op. /// create record for the same name is an idempotent no-op.
fn register_index_from_spec(self: *Engine, coll: *Collection, spec_doc: *const bson.Document) !void { fn register_index_from_spec(self: *Engine, coll: *Collection, spec_doc: *const bson.Document) !void {
@@ -721,6 +758,8 @@ fn apply_record(ctx: *anyopaque, record: storage.Record, doc: *bson.Document) an
self.live_docs += 1; self.live_docs += 1;
key_owned = true; key_owned = true;
stored = true; stored = true;
// The _id_ entry is added after replay, in build_all_indexes,
// together with the secondary indexes.
}, },
storage.record_type_delete => self.evict_doc(coll, id_key), storage.record_type_delete => self.evict_doc(coll, id_key),
else => {}, else => {},

293
src/index.zig
View File

@@ -1523,13 +1523,32 @@ pub const Plan = struct {
/// documents: the one covering the longest leading run of equality/$in /// documents: the one covering the longest leading run of equality/$in
/// predicates, optionally with a range on the next key. Returns null when /// predicates, optionally with a range on the next key. Returns null when
/// nothing usable remains — the caller scans. /// nothing usable remains — the caller scans.
pub fn plan(gpa: std.mem.Allocator, indexes: []const Index, filter: []const bson.Pair, sort: []const query.SortKey) !?Plan { ///
if (indexes.len == 0) return null; /// `id_ix` is the collection's implicit `_id_` index (kept separate from
/// the secondary `indexes` list so the listing/drop commands stay
/// unchanged). Every document has an `_id` and the index is not sparse, so
/// a full scan of it cannot miss a document — which is what the sort
/// planner's full-scan plan relies on. The encoded keys are canonical, so
/// `_id` equality on compare-equal values (int32 1, int64 1, double 1.0)
/// finds the same entries.
pub fn plan(
gpa: std.mem.Allocator,
id_ix: ?*const Index,
indexes: []const Index,
filter: []const bson.Pair,
sort: []const query.SortKey,
) !?Plan {
if (id_ix == null and indexes.len == 0) return null;
var clauses: std.ArrayListUnmanaged(Clause) = .empty; var clauses: std.ArrayListUnmanaged(Clause) = .empty;
defer clauses.deinit(gpa); defer clauses.deinit(gpa);
try flatten_clauses(gpa, filter, &clauses); try flatten_clauses(gpa, filter, &clauses);
var best: ?Plan = null; var best: ?Plan = null;
if (id_ix) |idx| {
if (try evaluate_index(gpa, idx, clauses.items, sort)) |cand| {
best = cand;
}
}
for (indexes) |*ix| { for (indexes) |*ix| {
var cand = (try evaluate_index(gpa, ix, clauses.items, sort)) orelse continue; var cand = (try evaluate_index(gpa, ix, clauses.items, sort)) orelse continue;
if (best) |b| { if (best) |b| {
@@ -1682,112 +1701,6 @@ fn evaluate_index(gpa: std.mem.Allocator, ix: *const Index, clauses: []const Cla
return pl; return pl;
} }
// ---------------------------------------------------------------------------
// _id fast path
// ---------------------------------------------------------------------------
pub const IdPlan = struct {
/// Values to look up in the docs map. A single $eq value is a heap copy
/// owned by this plan; a $in list aliases the filter's array. Freed by
/// deinit.
values: []const bson.Value,
owned: bool,
pub fn deinit(self: *IdPlan, gpa: std.mem.Allocator) void {
if (self.owned) gpa.free(self.values);
}
};
/// Plan for the implicit _id_ index (the docs map). Only applies to a
/// top-level _id equality/$eq/$in clause, and only when the queried value's
/// compare-equivalence class is serialization-canonical. bson.compare calls
/// int32 1, int64 1 and double 1.0 equal (and string/symbol/code "x" equal,
/// and nested variants), but serialize_value produces different map keys —
/// a hash lookup would then miss documents a scan would match.
///
/// The returned plan must be deinit'd: the single-value case is heap-copied
/// so the slice never points into a temporary (ReleaseFast reuses the stack,
/// which turned a dangling anonymous-list pointer into garbage).
pub fn plan_id(gpa: std.mem.Allocator, filter: []const bson.Pair) error{OutOfMemory}!?IdPlan {
// The first usable _id clause wins; no flattening buffer is needed
// because nothing is compared across clauses. $and members are searched
// like top-level pairs, every other operator skipped — same rule as
// flatten_clauses, and safe for the same reason (the full filter is
// re-applied to every candidate).
for (filter) |p| {
if (p.key.len > 0 and p.key[0] == '$') {
if (!std.mem.eql(u8, p.key, "$and")) continue;
const members = switch (p.value) {
.array => |a| a,
else => continue,
};
for (members) |m| {
const mp = switch (m) {
.doc => |d| d,
else => continue,
};
if (try plan_id(gpa, mp)) |found| return found;
}
continue;
}
if (!std.mem.eql(u8, p.key, "_id")) continue;
if (try id_lookup_values(gpa, p.value)) |lookup| return .{ .values = lookup.values, .owned = lookup.owned };
}
return null;
}
const IdLookup = struct {
values: []const bson.Value,
owned: bool,
};
/// The map-lookup values for one _id clause, or null when it is not a pure
/// equality/$in of fast-path-safe values. A range is unusable here: the docs
/// map is a hash, not an ordered structure.
fn id_lookup_values(gpa: std.mem.Allocator, v: bson.Value) error{OutOfMemory}!?IdLookup {
var info = CompInfo{};
analyze_clause(v, &info);
if (info.lo != null or info.hi != null) return null;
if (info.eq) |e| {
if (!value_fast_path_safe(e)) return null;
// Heap-copy the single value: a pointer to a stack or anonymous
// temporary would dangle once this frame returns.
const buf = try gpa.alloc(bson.Value, 1);
buf[0] = e;
return .{ .values = buf[0..1], .owned = true };
}
if (info.in_values) |list| {
for (list) |m| {
if (!value_fast_path_safe(m)) return null;
}
return .{ .values = list, .owned = false };
}
return null;
}
/// Whether serialize_value is injective on the value's compare-equivalence
/// class. False for numbers (int32 1 / int64 1 / double 1.0), string/symbol/
/// code (compared equal, serialized with different type tags), opaque
/// values (compared by payload only), and anything containing those.
fn value_fast_path_safe(v: bson.Value) bool {
return switch (v) {
.double, .int32, .int64, .string, .symbol, .code, .opaque_val => false,
.doc => |pairs| blk: {
for (pairs) |p| {
if (!value_fast_path_safe(p.value)) break :blk false;
}
break :blk true;
},
.array => |items| blk: {
for (items) |it| {
if (!value_fast_path_safe(it)) break :blk false;
}
break :blk true;
},
else => true,
};
}
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Tests // Tests
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -2301,63 +2214,6 @@ test "lookup_range matches a brute-force filter over random data" {
} }
} }
test "id fast path guards and $in" {
// plan_id may heap-copy the single value; run it through the allocator
// and free. The helper asserts on whether a plan was produced.
const gpa = testing.allocator;
const plans = struct {
fn has(pairs: []const bson.Pair) !bool {
var p = try plan_id(gpa, pairs);
defer if (p) |*pl| pl.deinit(gpa);
return p != null;
}
};
// Numbers never use the fast path (compare-equal but serialize-different).
try testing.expect(!try plans.has(&.{.{ .key = "_id", .value = .{ .int32 = 1 } }}));
// Strings are skipped too: a stored symbol/code _id compare-equals a
// string query but serializes differently.
try testing.expect(!try plans.has(&.{.{ .key = "_id", .value = .{ .string = "x" } }}));
// Truly canonical values (bool, ObjectId) do use it.
try testing.expect(try plans.has(&.{.{ .key = "_id", .value = .{ .bool = true } }}));
const oid = [_]u8{ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 };
try testing.expect(try plans.has(&.{.{ .key = "_id", .value = .{ .object_id = oid } }}));
// $in with a number member is skipped.
const mixed = [_]bson.Pair{.{ .key = "_id", .value = .{ .doc = &.{
.{ .key = "$in", .value = .{ .array = &.{ .{ .bool = true }, .{ .int32 = 1 } } } },
} } }};
try testing.expect(!try plans.has(&mixed));
const safe_in = [_]bson.Pair{.{ .key = "_id", .value = .{ .doc = &.{
.{ .key = "$in", .value = .{ .array = &.{ .{ .bool = true }, .{ .bool = false } } } },
} } }};
try testing.expect(try plans.has(&safe_in));
// $and members count as top-level.
const and_f = [_]bson.Pair{.{ .key = "$and", .value = .{ .array = &.{
.{ .doc = &.{.{ .key = "_id", .value = .{ .bool = true } }} },
} } }};
try testing.expect(try plans.has(&and_f));
// $or is not usable for the fast path.
const or_f = [_]bson.Pair{.{ .key = "$or", .value = .{ .array = &.{
.{ .doc = &.{.{ .key = "_id", .value = .{ .bool = true } }} },
} } }};
try testing.expect(!try plans.has(&or_f));
// A doc containing a number is not fast-path safe.
const doc_id = [_]bson.Pair{.{ .key = "_id", .value = .{ .doc = &.{.{ .key = "n", .value = .{ .int32 = 1 } }} } }};
try testing.expect(!try plans.has(&doc_id));
// A doc containing a string is unsafe too (string/symbol/code class).
const doc_str = [_]bson.Pair{.{ .key = "_id", .value = .{ .doc = &.{.{ .key = "n", .value = .{ .string = "x" } }} } }};
try testing.expect(!try plans.has(&doc_str));
// A doc of canonical values is safe.
const doc_safe = [_]bson.Pair{.{ .key = "_id", .value = .{ .doc = &.{.{ .key = "n", .value = .{ .bool = true } }} } }};
try testing.expect(try plans.has(&doc_safe));
// The single-value plan owns a usable copy: the ObjectId must read back
// exactly (this is the ReleaseFast regression the fix guards — a dangling
// pointer read back as garbage).
var p = try plan_id(gpa, &.{.{ .key = "_id", .value = .{ .object_id = oid } }});
defer if (p) |*pl| pl.deinit(gpa);
try testing.expect(p != null);
try testing.expectEqualSlices(u8, &oid, &p.?.values[0].object_id);
}
test "TTL spec round-trips through write_spec and compares in spec_equal" { test "TTL spec round-trips through write_spec and compares in spec_equal" {
const gpa = testing.allocator; const gpa = testing.allocator;
const spec = doc_of(&.{ const spec = doc_of(&.{
@@ -2471,6 +2327,93 @@ test "TTL spec rejects compound keys and bad expireAfterSeconds" {
try testing.expectEqualStrings("expireAt_1", ix.name); try testing.expectEqualStrings("expireAt_1", ix.name);
} }
test "the _id index plan covers equality, ranges and _id sort order" {
// The implicit _id_ index is a normal Index (keys = [_id: 1]) passed to
// plan separately from the secondaries. Its encoded keys are canonical,
// so compare-equal numbers (int32/int64/double) find the same entries,
// and a full scan of it is the sort planner's order supply for
// sort({_id: ...}).
const gpa = testing.allocator;
var ix = try simple_index(gpa, &.{"_id"}, false, false);
defer ix.deinit(gpa);
for (0..5) |i| {
const d = doc_of(&.{
.{ .key = "_id", .value = .{ .int32 = @intCast(i + 1) } },
.{ .key = "v", .value = .{ .int32 = @intCast(i) } },
});
const id = try std.fmt.allocPrint(gpa, "d{d}", .{i + 1});
defer gpa.free(id);
_ = try ix.add_doc(gpa, &d, id, false);
}
// {_id: 3} → an equality plan whose candidates are just that doc.
{
const f = [_]bson.Pair{.{ .key = "_id", .value = .{ .int32 = 3 } }};
var p = (try plan(gpa, &ix, &.{}, &f, &.{})).?;
defer p.deinit(gpa);
try testing.expectEqual(@as(usize, 1), p.key_len());
var ids: std.ArrayListUnmanaged([]const u8) = .empty;
defer ids.deinit(gpa);
try p.search(gpa, &ids);
try testing.expectEqual(@as(usize, 1), ids.items.len);
try testing.expectEqualStrings("d3", ids.items[0]);
}
// The same query as an int64 and a double finds the int32 entry.
{
const f = [_]bson.Pair{.{ .key = "_id", .value = .{ .int64 = 3 } }};
var p = (try plan(gpa, &ix, &.{}, &f, &.{})).?;
defer p.deinit(gpa);
var ids: std.ArrayListUnmanaged([]const u8) = .empty;
defer ids.deinit(gpa);
try p.search(gpa, &ids);
try testing.expectEqual(@as(usize, 1), ids.items.len);
try testing.expectEqualStrings("d3", ids.items[0]);
}
// A range on _id yields the band in index order.
{
const f = [_]bson.Pair{.{ .key = "_id", .value = .{ .doc = &.{.{ .key = "$gte", .value = .{ .int32 = 3 } }} } }};
var p = (try plan(gpa, &ix, &.{}, &f, &.{})).?;
defer p.deinit(gpa);
var ids: std.ArrayListUnmanaged([]const u8) = .empty;
defer ids.deinit(gpa);
try p.search(gpa, &ids);
try testing.expectEqual(@as(usize, 3), ids.items.len);
try testing.expectEqualStrings("d3", ids.items[0]);
try testing.expectEqualStrings("d5", ids.items[2]);
}
// sort({_id: 1}) with no filter: a full index scan that supplies the
// order — the plan the sort planner needs to stop materializing.
{
const sort = [_]query.SortKey{.{ .path = "_id", .descending = false }};
var p = (try plan(gpa, &ix, &.{}, &.{}, &sort)).?;
defer p.deinit(gpa);
try testing.expect(p.provides_sort and !p.backward);
var ids: std.ArrayListUnmanaged([]const u8) = .empty;
defer ids.deinit(gpa);
try p.search(gpa, &ids);
try testing.expectEqual(@as(usize, 5), ids.items.len);
try testing.expectEqualStrings("d1", ids.items[0]);
try testing.expectEqualStrings("d5", ids.items[4]);
}
// sort({_id: -1}): backward.
{
const sort = [_]query.SortKey{.{ .path = "_id", .descending = true }};
var p = (try plan(gpa, &ix, &.{}, &.{}, &sort)).?;
defer p.deinit(gpa);
try testing.expect(p.provides_sort and p.backward);
var ids: std.ArrayListUnmanaged([]const u8) = .empty;
defer ids.deinit(gpa);
try p.search(gpa, &ids);
try testing.expectEqualStrings("d5", ids.items[0]);
try testing.expectEqualStrings("d1", ids.items[4]);
}
// A filter naming no _id field leaves the _id index unusable.
{
const f = [_]bson.Pair{.{ .key = "v", .value = .{ .int32 = 1 } }};
try testing.expect((try plan(gpa, &ix, &.{}, &f, &.{})) == null);
}
}
test "planner picks eq run, ranges, and bails on sparse null" { test "planner picks eq run, ranges, and bails on sparse null" {
const gpa = testing.allocator; const gpa = testing.allocator;
var ix = try simple_index(gpa, &.{ "a", "b" }, false, false); var ix = try simple_index(gpa, &.{ "a", "b" }, false, false);
@@ -2484,7 +2427,7 @@ test "planner picks eq run, ranges, and bails on sparse null" {
.{ .key = "a", .value = .{ .int32 = 1 } }, .{ .key = "a", .value = .{ .int32 = 1 } },
.{ .key = "b", .value = .{ .int32 = 2 } }, .{ .key = "b", .value = .{ .int32 = 2 } },
}; };
var p = (try plan(gpa, &.{ix}, &f, &.{})).?; var p = (try plan(gpa, null, &.{ix}, &f, &.{})).?;
defer p.deinit(gpa); defer p.deinit(gpa);
try testing.expectEqual(@as(usize, 2), p.key_len()); try testing.expectEqual(@as(usize, 2), p.key_len());
try testing.expect(p.lo == null and p.hi == null); try testing.expect(p.lo == null and p.hi == null);
@@ -2495,7 +2438,7 @@ test "planner picks eq run, ranges, and bails on sparse null" {
.{ .key = "a", .value = .{ .int32 = 1 } }, .{ .key = "a", .value = .{ .int32 = 1 } },
.{ .key = "b", .value = .{ .doc = &.{.{ .key = "$gt", .value = .{ .int32 = 2 } }} } }, .{ .key = "b", .value = .{ .doc = &.{.{ .key = "$gt", .value = .{ .int32 = 2 } }} } },
}; };
var p = (try plan(gpa, &.{ix}, &f, &.{})).?; var p = (try plan(gpa, null, &.{ix}, &f, &.{})).?;
defer p.deinit(gpa); defer p.deinit(gpa);
try testing.expectEqual(@as(usize, 1), p.key_len()); try testing.expectEqual(@as(usize, 1), p.key_len());
try testing.expect(p.hi == null and p.lo != null and !p.lo_incl); try testing.expect(p.hi == null and p.lo != null and !p.lo_incl);
@@ -2503,14 +2446,14 @@ test "planner picks eq run, ranges, and bails on sparse null" {
// {a: 1} only → prefix run of 1. // {a: 1} only → prefix run of 1.
{ {
const f = [_]bson.Pair{.{ .key = "a", .value = .{ .int32 = 1 } }}; const f = [_]bson.Pair{.{ .key = "a", .value = .{ .int32 = 1 } }};
var p = (try plan(gpa, &.{ix}, &f, &.{})).?; var p = (try plan(gpa, null, &.{ix}, &f, &.{})).?;
defer p.deinit(gpa); defer p.deinit(gpa);
try testing.expectEqual(@as(usize, 1), p.key_len()); try testing.expectEqual(@as(usize, 1), p.key_len());
} }
// Pure range on the first key → key_len 0 with a bound. // Pure range on the first key → key_len 0 with a bound.
{ {
const f = [_]bson.Pair{.{ .key = "a", .value = .{ .doc = &.{.{ .key = "$gte", .value = .{ .int32 = 1 } }} } }}; const f = [_]bson.Pair{.{ .key = "a", .value = .{ .doc = &.{.{ .key = "$gte", .value = .{ .int32 = 1 } }} } }};
var p = (try plan(gpa, &.{ix}, &f, &.{})).?; var p = (try plan(gpa, null, &.{ix}, &f, &.{})).?;
defer p.deinit(gpa); defer p.deinit(gpa);
try testing.expectEqual(@as(usize, 0), p.key_len()); try testing.expectEqual(@as(usize, 0), p.key_len());
try testing.expect(p.lo != null and p.lo_incl); try testing.expect(p.lo != null and p.lo_incl);
@@ -2518,23 +2461,23 @@ test "planner picks eq run, ranges, and bails on sparse null" {
// Unusable filter → no plan. // Unusable filter → no plan.
{ {
const f = [_]bson.Pair{.{ .key = "a", .value = .{ .doc = &.{.{ .key = "$regex", .value = .{ .string = "^x" } }} } }}; const f = [_]bson.Pair{.{ .key = "a", .value = .{ .doc = &.{.{ .key = "$regex", .value = .{ .string = "^x" } }} } }};
try testing.expect((try plan(gpa, &.{ix}, &f, &.{})) == null); try testing.expect((try plan(gpa, null, &.{ix}, &f, &.{})) == null);
const or_f = [_]bson.Pair{.{ .key = "$or", .value = .{ .array = &.{ const or_f = [_]bson.Pair{.{ .key = "$or", .value = .{ .array = &.{
.{ .doc = &.{.{ .key = "a", .value = .{ .int32 = 1 } }} }, .{ .doc = &.{.{ .key = "a", .value = .{ .int32 = 1 } }} },
} } }}; } } }};
try testing.expect((try plan(gpa, &.{ix}, &or_f, &.{})) == null); try testing.expect((try plan(gpa, null, &.{ix}, &or_f, &.{})) == null);
} }
// Sparse index bails on a null component. // Sparse index bails on a null component.
{ {
const f = [_]bson.Pair{.{ .key = "a", .value = .null }}; const f = [_]bson.Pair{.{ .key = "a", .value = .null }};
try testing.expect((try plan(gpa, &.{sp}, &f, &.{})) == null); try testing.expect((try plan(gpa, null, &.{sp}, &f, &.{})) == null);
// Non-sparse is fine with null. // Non-sparse is fine with null.
var p = (try plan(gpa, &.{ix}, &f, &.{})).?; var p = (try plan(gpa, null, &.{ix}, &f, &.{})).?;
defer p.deinit(gpa); defer p.deinit(gpa);
try testing.expect(p.key_len() == 1); try testing.expect(p.key_len() == 1);
// A null inside $in bails too. // A null inside $in bails too.
const fin = [_]bson.Pair{.{ .key = "a", .value = .{ .doc = &.{.{ .key = "$in", .value = .{ .array = &.{ .{ .int32 = 1 }, .null } } }} } }}; const fin = [_]bson.Pair{.{ .key = "a", .value = .{ .doc = &.{.{ .key = "$in", .value = .{ .array = &.{ .{ .int32 = 1 }, .null } } }} } }};
try testing.expect((try plan(gpa, &.{sp}, &fin, &.{})) == null); try testing.expect((try plan(gpa, null, &.{sp}, &fin, &.{})) == null);
} }
// $in cartesian product is capped. // $in cartesian product is capped.
{ {
@@ -2545,6 +2488,6 @@ test "planner picks eq run, ranges, and bails on sparse null" {
.{ .key = "b", .value = .{ .doc = &.{.{ .key = "$in", .value = .{ .array = &members } }} } }, .{ .key = "b", .value = .{ .doc = &.{.{ .key = "$in", .value = .{ .array = &members } }} } },
}; };
// 20 * 20 = 400 > 100 → fall back to a scan. // 20 * 20 = 400 > 100 → fall back to a scan.
try testing.expect((try plan(gpa, &.{ix}, &f, &.{})) == null); try testing.expect((try plan(gpa, null, &.{ix}, &f, &.{})) == null);
} }
} }

38
tests/e2e/results/phase3.txt Normal file
View File

@@ -0,0 +1,38 @@
# Phase 3 gate — mongo-lite vs MongoDB 8.3.7, 1g dataset / ~16k docs
# Ratio < 1.0 = mongo-lite faster. Reproduce: bash tests/e2e/compare-run.sh 1g 16k
# This run includes roadmap items 1 (B+tree) and 2 (ordered _id index).
# Compare: tests/e2e/results/phase1.txt (pre-tree baseline).
benchmark mongo-lite mongodb ratio
insertOne (sequential) ×200 0.19 ms 4.4 ms 0.0x
bulk insert throughput 816.5 MB/s 716.8 MB/s 1.1x
docs loaded 65,536 65,536 1.0x
createIndex({k: 1}) 50.7 ms 76.8 ms 0.7x
countDocuments({}) 1.5 ms 11.1 ms 0.1x
findOne({_id: <ObjectId>}) 0.57 ms 0.64 ms 0.9x
findOne({k: 500}) (indexed) 0.58 ms 1.5 ms 0.4x
find({p: {$gte,$lt}}).count() (scan) 22.1 ms 12.9 ms 1.7x
find({}).sort({_id:-1}).limit(20) 2.4 ms 2.2 ms 1.1x
find({}, {proj}).limit(1000) 3.7 ms 4.3 ms 0.9x
aggregate $group by k 9.1 ms 12.5 ms 0.7x
updateOne({_id}) ×50 0.16 ms 0.19 ms 0.8x
updateMany({k: 7}, {$inc}) 1.6 ms 6.4 ms 0.3x
deleteOne({_id}) + insertOne 0.62 ms 4.8 ms 0.1x
node client RSS 159 MB 156 MB 1.0x
server RSS 1973 MB 1334 MB
kill -9 reopen 0.8s 1.3s
db on disk 1025MB 91MB
# Item 2 (ordered _id index) deltas vs phase2:
# sort({_id:-1}).limit(20) 6.2 -> 2.4 ms (2.3x slower than mongod -> parity):
# the sort planner now scans the _id tree in order and stops
# at the page limit instead of materializing every candidate.
# integer/string _id findOne, updateOne, deleteOne no longer fall back to a
# full collection scan (verified separately: point lookups,
# $in, ranges, int64/int32 compare-equal equality all hit the
# tree; the docs-map serialization-guard fast path is gone).
#
# Remaining gaps and where they are addressed:
# db on disk 11x -> Phase 3 (block-compressed log)
# range-scan 1.7x -> Phase 4 (contiguous byte storage, not the matcher)
# server RSS 1.5x -> Phase 4 (per-document arena -> byte storage)