dev/MultiforaDB
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

index: ordered _id index (roadmap item 2)

Browse Source 
Give every Collection an implicit _id_ index (a normal Index with keys
[_id: 1]) so _id equality, $in, ranges and sorts stop depending on the
docs-map hash or a full scan. Kept out of the secondary indexes list, so
listIndexes/dropIndexes/createIndex and the log format are unchanged (no
index_create record, no double listing) and e2e3.js passes unmodified.

Maintained in upsert through the same reserve-then-insert protocol as
the secondaries, removed in evict_doc, and rebuilt after replay by
build_all_indexes alongside them (never maintained mid-replay, so a
failed add can't leave the index under-approximating). index.plan now
takes it as a separate argument. Its keys are canonical
(bson.encode_key gives int32 1, int64 1 and double 1.0 identical bytes),
so the serialization-guarded docs-map fast path (plan_id,
value_fast_path_safe and friends) is deleted.

Measured (tests/e2e/results/phase3.txt): sort({_id:-1}).limit(20) 6.2 ->
2.4 ms (2.3x slower than MongoDB -> parity); integer/string _id point
lookups, $in and ranges verified against the tree. Unit suite in all
three optimize modes, the crash pair, e2e3/e2e4/e2e6.
This commit is contained in:
Alexey Shakhmatov
2026-08-02 21:18:40 +03:00
parent 61fe952125
commit 58914a69c3
6 changed files with 267 additions and 245 deletions
Download Patch File Download Diff File Expand all files Collapse all files

39
README.md
View File

@@ -190,14 +190,14 @@ With the ReleaseFast default build (MongoDB 8.3.7 on the same Mac):
| createIndex({k: 1}) | 51 ms | 82 ms | **mongo-lite** |
| countDocuments({}) | 1.5 ms | 13.8 ms | **mongo-lite ×9** |
| findOne({_id}) | 0.57 ms | 0.67 ms | mongo-lite |
| findOne indexed | 0.57 ms | 1.8 ms | **mongo-lite ×3** |
| range-scan count | 20 ms | 13 ms | mongodb ×1.6 |
| sort + limit(20), on `_id` | 6.2 ms | 2.7 ms | mongodb ×2.3 |
| findOne indexed | 0.58 ms | 1.5 ms | **mongo-lite ×2.6** |
| range-scan count | 22 ms | 13 ms | mongodb ×1.7 |
| sort + limit(20), on `_id` | 2.4 ms | 2.2 ms | mongodb ×1.1 |
| sort + limit(20), indexed field | 1.0 ms | — | — |
| aggregate $group | 11.5 ms | 15.5 ms | **mongo-lite** |
| updateOne({_id}) | 0.17 ms | 0.19 ms | mongo-lite |
| updateMany (65 docs) | 1.8 ms | 6.7 ms | **mongo-lite ×3.7** |
| deleteOne + insert | 0.50 ms | 5.0 ms | **mongo-lite ×10** |
| updateMany (65 docs) | 1.6 ms | 6.4 ms | **mongo-lite ×4** |
| deleteOne + insert | 0.62 ms | 4.8 ms | **mongo-lite ×8** |
| server RSS | 2.0 GB | 1.5 GB | mongodb (×0.7) |
| kill -9 → reopen | 0.8 s | 1.3 s | **mongo-lite** |
| db on disk | 1.0 GB | 96 MB | mongodb (compressed) |
@@ -206,15 +206,12 @@ The remaining losses are structural rather than incidental. Disk size is
the big one: payloads are stored raw, so the log is 11x MongoDB's
compressed files. RSS trails because every document carries its own arena.
The range-scan gap is not the matcher — it is walking 65,536 documents
that each live in a separate allocation, one pointer chase apiece. And
`sort` on `_id` still materializes candidates because nothing ordered
covers `_id` yet; the same sort on an indexed field streams straight out
of the index at 1.0 ms.
that each live in a separate allocation, one pointer chase apiece.
Reproduce the table with `bash tests/e2e/compare-run.sh 1g 16k`; the
pre-tree baseline is recorded in `tests/e2e/results/phase1.txt`, the run
above (with the B+tree, roadmap item 1) in
`tests/e2e/results/phase2.txt`.
pre-tree baseline is recorded in `tests/e2e/results/phase1.txt`, and the
runs with the B+tree and ordered `_id` index (roadmap items 1 and 2) in
`tests/e2e/results/phase2.txt` and `tests/e2e/results/phase3.txt`.
### What is left (highest impact first)
@@ -226,15 +223,11 @@ traps in [ROADMAP.md](ROADMAP.md).
highly compressible workloads massively; note Zig 0.16 ships zstd
decompression only, and deflate would cap writes below the current
insert rate.
2. **An ordered `_id` index** — `sort({_id: ...})` still materializes every
candidate, and integer `_id`s still scan. Both fall out of indexing the
encoded `_id`. The tree is in place, so an `_id` index update is a
leaf insert, not a tail memmove.
3. **Stop giving every document its own arena** — the source of both the
2. **Stop giving every document its own arena** — the source of both the
RSS gap and the range-scan gap. Storing canonical BSON bytes in a
per-collection slab and matching against them (parsing only the fields a
filter names) makes scans contiguous instead of a pointer chase.
4. **Decompose the global lock** — one reader/writer lock covers the whole
3. **Decompose the global lock** — one reader/writer lock covers the whole
engine and is held across fsync, compaction and reply construction.
Per-collection locks plus cross-connection group commit are the path to
using more than one core on writes.
@@ -257,8 +250,16 @@ Done so far, with the measurement that drove each:
records, no rebalancing on delete, and bulk bottom-up packing. Entry
insertion and removal are a descent plus a leaf-local edit instead of a
tail memmove, so writes into an already-built index stopped being
quadratic. `updateMany` 17.3 → 1.8 ms (2.8x slower than MongoDB → 3.7x
quadratic. `updateMany` 17.3 → 1.6 ms (2.8x slower than MongoDB → 4x
faster); `createIndex` 62 → 51 ms.
- **An ordered `_id` index** (roadmap item 2): every collection carries an
implicit `_id_` index (kept out of the secondary list, so the listing,
drop and log-format surfaces are unchanged; rebuilt after replay like
the secondaries). Its encoded keys are canonical, so the old
serialization-guarded docs-map fast path is gone and integer/string
`_id` point lookups, `$in` and ranges hit the tree instead of a full
scan. `sort({_id: ...})` is now an index-ordered scan with an early stop:
`sort+limit(20)` 6.2 → 2.4 ms (parity with MongoDB).
- **Entry removal is a binary search**, not a scan of the whole index.
`updateMany` 15.4 → 5.5 ms.
- **Top-k sort selection** and an allocation-free decorate pass, plus

38
ROADMAP.md
View File

@@ -1,9 +1,10 @@
# Remaining performance work
Status: **item 1 (B+tree over the encoded keys) is done** — landed and
verified in `tests/e2e/results/phase2.txt` (updateMany 17.3 → 1.8 ms,
createIndex 62 → 51 ms). Its dependents (items 2 and 4) now stand on a
tree instead of a sorted array. Five items below, in dependency order.
Status: **items 1 (B+tree over the encoded keys) and 2 (ordered `_id` index)
are done** — landed and verified in `tests/e2e/results/phase2.txt` and
`phase3.txt` (updateMany 17.3 → 1.6 ms, createIndex 62 → 51 ms, `_id`
sort+limit 6.2 → 2.4 ms). Their dependents (items 4) now stand on a tree
instead of a sorted array. Items below, in dependency order.
Each is sized to be landed and verified on
its own; the ordering constraints between them are the load-bearing part, so
read those before picking one up.
@@ -101,7 +102,26 @@ differential. Both already exist and both are mutation-checked.
---
## 2. Ordered `_id` index — **depends on 1**
## 2. Ordered `_id` index — DONE
Landed as an implicit `_id_` index on every `Collection` (a normal
`index.Index` with keys `[_id: 1]`, kept out of the secondary `indexes`
list so `listIndexes`/`dropIndexes`/`createIndex` and the log format are
unchanged — no `index_create` record, no double listing). Maintained in
`upsert` (through the same reserve-then-insert protocol as the
secondaries) and `evict_doc`; rebuilt after replay by `build_all_indexes`
alongside the secondaries. `index.plan` now takes it as a separate
argument, so `{_id: ...}` equality, `$in` and ranges use the tree (the
old serialization-guarded docs-map fast path — `plan_id`,
`value_fast_path_safe` and friends — is deleted), and `sort({_id: ...})`
becomes an index-ordered full scan with an early stop. A full `_id` scan
cannot miss a document: every doc has an `_id` and the index is not
sparse, and its keys are canonical (`bson.encode_key` gives int32 1,
int64 1 and double 1.0 identical bytes).
Recorded deltas vs `tests/e2e/results/phase2.txt`: `sort({_id:-1}).limit(20)`
6.2 → 2.4 ms (2.3x slower than MongoDB → parity). Integer/string `_id`
point lookups, `$in` and ranges no longer full-scan.
**Why.** `sort({_id: ...})` still materializes every candidate, and integer
`_id`s still fall back to a full collection scan on every `findOne`,
@@ -115,8 +135,8 @@ requires.
Then delete `value_fast_path_safe` and friends (`src/index.zig`). They exist
only because `serialize_value` gives `int32 1`, `int64 1` and `double 1.0`
different bytes despite comparing equal. `bson.encode_key` already gives them
identical bytes, so the guard is obsolete.
different bytes despite comparing equal. `bson.encode_key` already gives
them identical bytes, so the guard is obsolete.
**Why it depends on item 1.** This index updates on *every* insert. Against a
sorted array that is a tail memmove each time — roughly 51 GB of memmove over
@@ -128,7 +148,9 @@ only" caveat.
**Watch.** A real `_id_` index may start appearing in `listIndexes` and
writing an `index_create` record to the log. `e2e3.js` asserts on index
listings — check it before assuming this is invisible.
listings — check it before assuming this is invisible. (This landed without
either: the index stays out of the secondary list, so the listing, drop and
log surfaces are untouched; verified with `e2e3.js` unchanged.)
---

29
src/commands.zig
View File

@@ -644,31 +644,10 @@ fn scan_sorted(
const filter_doc = bson.Document{ .arena = undefined, .pairs = filter };
var n: usize = 0;
// _id_ fast path: the docs map is the _id index. Skipped when the
// queried value's compare-equivalence class is serialization-ambiguous
// (see index.plan_id).
if (try index.plan_id(ctx.gpa, filter)) |id_plan| {
var plan = id_plan;
defer plan.deinit(ctx.gpa);
// One scratch key, rebuilt per value: a key is never needed past its
// own lookup.
var key: std.ArrayListUnmanaged(u8) = .empty;
defer key.deinit(ctx.gpa);
for (plan.values) |v| {
key.clearRetainingCapacity();
try bson.write_serialized_value(v, ctx.gpa, &key);
const doc = coll.docs.get(key.items) orelse continue;
if (!try query.matches(ctx.gpa, &filter_doc, doc)) continue;
if (out) |list| try list.append(ctx.gpa, doc);
n += 1;
if (lim != 0 and n >= lim) break;
}
return n;
}
// Secondary-index plan: candidates in index order, re-filtered. The
// returned ids alias the docs map keys, valid under the read lock.
if (try index.plan(ctx.gpa, coll.indexes.items, filter, sort)) |p| {
// Index plan (the implicit _id_ index first, then the secondaries):
// candidates in index order, re-filtered. The returned ids alias the
// docs map keys, valid under the read lock.
if (try index.plan(ctx.gpa, &coll.id_index, coll.indexes.items, filter, sort)) |p| {
var plan = p;
defer plan.deinit(ctx.gpa);
var ids: std.ArrayListUnmanaged([]const u8) = .empty;

55
src/db.zig
View File

@@ -12,10 +12,23 @@ const index = @import("index.zig");
pub const Collection = struct {
docs: std.StringHashMapUnmanaged(*bson.Document),
/// Secondary indexes (persisted through the log).
indexes: std.ArrayListUnmanaged(index.Index),
/// The implicit _id_ index: every document has an _id and it is not
/// sparse, so entry count equals document count and a full scan of it
/// cannot miss a document — which is what the sort planner's full-scan
/// plan relies on. Kept out of `indexes` so the listing/drop commands
/// and the log format are unchanged (it is rebuilt on open like
/// everything else). `bson.encode_key` keys are canonical, so it also
/// replaces the old serialization-guarded docs-map fast path for
/// integer/string/etc. _id lookups.
id_index: index.Index,
fn init() Collection {
return .{ .docs = .empty, .indexes = .empty };
fn init(gpa: std.mem.Allocator) !Collection {
var self: Collection = .{ .docs = .empty, .indexes = .empty, .id_index = undefined };
const keys = [_]index.IndexKey{.{ .path = "_id", .descending = false }};
self.id_index = try index.Index.init(gpa, "_id_", &keys, false, false, null);
return self;
}
/// The secondary index with this name, or null. The single by-name
@@ -108,6 +121,7 @@ pub const Engine = struct {
// Dropping a collection turns all of its records into garbage.
self.live_docs -= coll.docs.count();
self.dead_docs += coll.docs.count();
coll.id_index.deinit(self.gpa);
for (coll.indexes.items) |*ix| ix.deinit(self.gpa);
coll.indexes.deinit(self.gpa);
var doc_it = coll.docs.iterator();
@@ -139,6 +153,7 @@ pub const Engine = struct {
/// regenerating them from it, which is far cheaper than scanning.
fn evict_doc(self: *Engine, coll: *Collection, id_key: []const u8) void {
const old = coll.docs.fetchRemove(id_key) orelse return;
coll.id_index.remove_doc(self.gpa, old.value, old.key);
for (coll.indexes.items) |*ix| ix.remove_doc(self.gpa, old.value, old.key);
old.value.*.deinit();
self.gpa.destroy(old.value);
@@ -241,6 +256,15 @@ pub const Engine = struct {
return err;
};
}
{
// The implicit _id_ index, through the same protocol: reserved
// before the log append, inserted infallibly after it.
var built = try coll.id_index.build_entries(self.gpa, owned, id_key);
built_list.append(self.gpa, .{ .built = built, .ix = &coll.id_index }) catch |err| {
built.deinit(self.gpa);
return err;
};
}
// 2. The _id check, mirroring the pre-index behavior.
if (mode == .insert and coll.docs.contains(id_key)) return error.DuplicateKey;
@@ -494,7 +518,9 @@ pub const Engine = struct {
if (db.collections.getPtr(coll_name)) |coll| return coll;
const coll_key = try self.gpa.dupe(u8, coll_name);
errdefer self.gpa.free(coll_key);
try db.collections.put(self.gpa, coll_key, Collection.init());
var new_coll = try Collection.init(self.gpa);
errdefer new_coll.id_index.deinit(self.gpa);
try db.collections.put(self.gpa, coll_key, new_coll);
return db.collections.getPtr(coll_name) orelse unreachable;
}
@@ -624,8 +650,22 @@ pub const Engine = struct {
var coll_it = db_entry.value_ptr.collections.iterator();
while (coll_it.next()) |coll_entry| {
for (coll_entry.value_ptr.indexes.items) |*ix| {
if (ix.count() > 0) continue; // defensive
var doc_it = coll_entry.value_ptr.docs.iterator();
try self.rebuild_index(coll_entry.value_ptr, ix);
}
try self.rebuild_index(coll_entry.value_ptr, &coll_entry.value_ptr.id_index);
}
}
}
/// Rebuild one index from the live documents. Runs after replay, so it
/// is order-independent; indexes already holding entries (maintained
/// live) are skipped defensively. A duplicate under a unique index logs
/// a loud warning and keeps the index (still correct as a candidate
/// generator; future writes are still enforced) — the database always
/// opens, leaving dropIndexes as an in-band recovery path.
fn rebuild_index(self: *Engine, coll: *Collection, ix: *index.Index) !void {
if (ix.count() > 0) return; // defensive
var doc_it = coll.docs.iterator();
while (doc_it.next()) |doc_entry| {
ix.append_doc_entries(self.gpa, doc_entry.value_ptr.*, doc_entry.key_ptr.*) catch |err| switch (err) {
error.ParallelArrays => {
@@ -640,9 +680,6 @@ pub const Engine = struct {
std.debug.print("mongo-lite: WARNING: unique index '{s}' has duplicate keys in existing data; duplicates not enforced for existing documents\n", .{ix.name});
}
}
}
}
}
/// Register an (empty) index from a persisted spec document. A repeated
/// create record for the same name is an idempotent no-op.
@@ -721,6 +758,8 @@ fn apply_record(ctx: *anyopaque, record: storage.Record, doc: *bson.Document) an
self.live_docs += 1;
key_owned = true;
stored = true;
// The _id_ entry is added after replay, in build_all_indexes,
// together with the secondary indexes.
},
storage.record_type_delete => self.evict_doc(coll, id_key),
else => {},

293
src/index.zig
View File

@@ -1523,13 +1523,32 @@ pub const Plan = struct {
/// documents: the one covering the longest leading run of equality/$in
/// predicates, optionally with a range on the next key. Returns null when
/// nothing usable remains — the caller scans.
pub fn plan(gpa: std.mem.Allocator, indexes: []const Index, filter: []const bson.Pair, sort: []const query.SortKey) !?Plan {
if (indexes.len == 0) return null;
///
/// `id_ix` is the collection's implicit `_id_` index (kept separate from
/// the secondary `indexes` list so the listing/drop commands stay
/// unchanged). Every document has an `_id` and the index is not sparse, so
/// a full scan of it cannot miss a document — which is what the sort
/// planner's full-scan plan relies on. The encoded keys are canonical, so
/// `_id` equality on compare-equal values (int32 1, int64 1, double 1.0)
/// finds the same entries.
pub fn plan(
gpa: std.mem.Allocator,
id_ix: ?*const Index,
indexes: []const Index,
filter: []const bson.Pair,
sort: []const query.SortKey,
) !?Plan {
if (id_ix == null and indexes.len == 0) return null;
var clauses: std.ArrayListUnmanaged(Clause) = .empty;
defer clauses.deinit(gpa);
try flatten_clauses(gpa, filter, &clauses);
var best: ?Plan = null;
if (id_ix) |idx| {
if (try evaluate_index(gpa, idx, clauses.items, sort)) |cand| {
best = cand;
}
}
for (indexes) |*ix| {
var cand = (try evaluate_index(gpa, ix, clauses.items, sort)) orelse continue;
if (best) |b| {
@@ -1682,112 +1701,6 @@ fn evaluate_index(gpa: std.mem.Allocator, ix: *const Index, clauses: []const Cla
return pl;
}
// ---------------------------------------------------------------------------
// _id fast path
// ---------------------------------------------------------------------------
pub const IdPlan = struct {
/// Values to look up in the docs map. A single $eq value is a heap copy
/// owned by this plan; a $in list aliases the filter's array. Freed by
/// deinit.
values: []const bson.Value,
owned: bool,
pub fn deinit(self: *IdPlan, gpa: std.mem.Allocator) void {
if (self.owned) gpa.free(self.values);
}
};
/// Plan for the implicit _id_ index (the docs map). Only applies to a
/// top-level _id equality/$eq/$in clause, and only when the queried value's
/// compare-equivalence class is serialization-canonical. bson.compare calls
/// int32 1, int64 1 and double 1.0 equal (and string/symbol/code "x" equal,
/// and nested variants), but serialize_value produces different map keys —
/// a hash lookup would then miss documents a scan would match.
///
/// The returned plan must be deinit'd: the single-value case is heap-copied
/// so the slice never points into a temporary (ReleaseFast reuses the stack,
/// which turned a dangling anonymous-list pointer into garbage).
pub fn plan_id(gpa: std.mem.Allocator, filter: []const bson.Pair) error{OutOfMemory}!?IdPlan {
// The first usable _id clause wins; no flattening buffer is needed
// because nothing is compared across clauses. $and members are searched
// like top-level pairs, every other operator skipped — same rule as
// flatten_clauses, and safe for the same reason (the full filter is
// re-applied to every candidate).
for (filter) |p| {
if (p.key.len > 0 and p.key[0] == '$') {
if (!std.mem.eql(u8, p.key, "$and")) continue;
const members = switch (p.value) {
.array => |a| a,
else => continue,
};
for (members) |m| {
const mp = switch (m) {
.doc => |d| d,
else => continue,
};
if (try plan_id(gpa, mp)) |found| return found;
}
continue;
}
if (!std.mem.eql(u8, p.key, "_id")) continue;
if (try id_lookup_values(gpa, p.value)) |lookup| return .{ .values = lookup.values, .owned = lookup.owned };
}
return null;
}
const IdLookup = struct {
values: []const bson.Value,
owned: bool,
};
/// The map-lookup values for one _id clause, or null when it is not a pure
/// equality/$in of fast-path-safe values. A range is unusable here: the docs
/// map is a hash, not an ordered structure.
fn id_lookup_values(gpa: std.mem.Allocator, v: bson.Value) error{OutOfMemory}!?IdLookup {
var info = CompInfo{};
analyze_clause(v, &info);
if (info.lo != null or info.hi != null) return null;
if (info.eq) |e| {
if (!value_fast_path_safe(e)) return null;
// Heap-copy the single value: a pointer to a stack or anonymous
// temporary would dangle once this frame returns.
const buf = try gpa.alloc(bson.Value, 1);
buf[0] = e;
return .{ .values = buf[0..1], .owned = true };
}
if (info.in_values) |list| {
for (list) |m| {
if (!value_fast_path_safe(m)) return null;
}
return .{ .values = list, .owned = false };
}
return null;
}
/// Whether serialize_value is injective on the value's compare-equivalence
/// class. False for numbers (int32 1 / int64 1 / double 1.0), string/symbol/
/// code (compared equal, serialized with different type tags), opaque
/// values (compared by payload only), and anything containing those.
fn value_fast_path_safe(v: bson.Value) bool {
return switch (v) {
.double, .int32, .int64, .string, .symbol, .code, .opaque_val => false,
.doc => |pairs| blk: {
for (pairs) |p| {
if (!value_fast_path_safe(p.value)) break :blk false;
}
break :blk true;
},
.array => |items| blk: {
for (items) |it| {
if (!value_fast_path_safe(it)) break :blk false;
}
break :blk true;
},
else => true,
};
}
// ---------------------------------------------------------------------------
// Tests
// ---------------------------------------------------------------------------
@@ -2301,63 +2214,6 @@ test "lookup_range matches a brute-force filter over random data" {
}
}
test "id fast path guards and $in" {
// plan_id may heap-copy the single value; run it through the allocator
// and free. The helper asserts on whether a plan was produced.
const gpa = testing.allocator;
const plans = struct {
fn has(pairs: []const bson.Pair) !bool {
var p = try plan_id(gpa, pairs);
defer if (p) |*pl| pl.deinit(gpa);
return p != null;
}
};
// Numbers never use the fast path (compare-equal but serialize-different).
try testing.expect(!try plans.has(&.{.{ .key = "_id", .value = .{ .int32 = 1 } }}));
// Strings are skipped too: a stored symbol/code _id compare-equals a
// string query but serializes differently.
try testing.expect(!try plans.has(&.{.{ .key = "_id", .value = .{ .string = "x" } }}));
// Truly canonical values (bool, ObjectId) do use it.
try testing.expect(try plans.has(&.{.{ .key = "_id", .value = .{ .bool = true } }}));
const oid = [_]u8{ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 };
try testing.expect(try plans.has(&.{.{ .key = "_id", .value = .{ .object_id = oid } }}));
// $in with a number member is skipped.
const mixed = [_]bson.Pair{.{ .key = "_id", .value = .{ .doc = &.{
.{ .key = "$in", .value = .{ .array = &.{ .{ .bool = true }, .{ .int32 = 1 } } } },
} } }};
try testing.expect(!try plans.has(&mixed));
const safe_in = [_]bson.Pair{.{ .key = "_id", .value = .{ .doc = &.{
.{ .key = "$in", .value = .{ .array = &.{ .{ .bool = true }, .{ .bool = false } } } },
} } }};
try testing.expect(try plans.has(&safe_in));
// $and members count as top-level.
const and_f = [_]bson.Pair{.{ .key = "$and", .value = .{ .array = &.{
.{ .doc = &.{.{ .key = "_id", .value = .{ .bool = true } }} },
} } }};
try testing.expect(try plans.has(&and_f));
// $or is not usable for the fast path.
const or_f = [_]bson.Pair{.{ .key = "$or", .value = .{ .array = &.{
.{ .doc = &.{.{ .key = "_id", .value = .{ .bool = true } }} },
} } }};
try testing.expect(!try plans.has(&or_f));
// A doc containing a number is not fast-path safe.
const doc_id = [_]bson.Pair{.{ .key = "_id", .value = .{ .doc = &.{.{ .key = "n", .value = .{ .int32 = 1 } }} } }};
try testing.expect(!try plans.has(&doc_id));
// A doc containing a string is unsafe too (string/symbol/code class).
const doc_str = [_]bson.Pair{.{ .key = "_id", .value = .{ .doc = &.{.{ .key = "n", .value = .{ .string = "x" } }} } }};
try testing.expect(!try plans.has(&doc_str));
// A doc of canonical values is safe.
const doc_safe = [_]bson.Pair{.{ .key = "_id", .value = .{ .doc = &.{.{ .key = "n", .value = .{ .bool = true } }} } }};
try testing.expect(try plans.has(&doc_safe));
// The single-value plan owns a usable copy: the ObjectId must read back
// exactly (this is the ReleaseFast regression the fix guards — a dangling
// pointer read back as garbage).
var p = try plan_id(gpa, &.{.{ .key = "_id", .value = .{ .object_id = oid } }});
defer if (p) |*pl| pl.deinit(gpa);
try testing.expect(p != null);
try testing.expectEqualSlices(u8, &oid, &p.?.values[0].object_id);
}
test "TTL spec round-trips through write_spec and compares in spec_equal" {
const gpa = testing.allocator;
const spec = doc_of(&.{
@@ -2471,6 +2327,93 @@ test "TTL spec rejects compound keys and bad expireAfterSeconds" {
try testing.expectEqualStrings("expireAt_1", ix.name);
}
test "the _id index plan covers equality, ranges and _id sort order" {
// The implicit _id_ index is a normal Index (keys = [_id: 1]) passed to
// plan separately from the secondaries. Its encoded keys are canonical,
// so compare-equal numbers (int32/int64/double) find the same entries,
// and a full scan of it is the sort planner's order supply for
// sort({_id: ...}).
const gpa = testing.allocator;
var ix = try simple_index(gpa, &.{"_id"}, false, false);
defer ix.deinit(gpa);
for (0..5) |i| {
const d = doc_of(&.{
.{ .key = "_id", .value = .{ .int32 = @intCast(i + 1) } },
.{ .key = "v", .value = .{ .int32 = @intCast(i) } },
});
const id = try std.fmt.allocPrint(gpa, "d{d}", .{i + 1});
defer gpa.free(id);
_ = try ix.add_doc(gpa, &d, id, false);
}
// {_id: 3} → an equality plan whose candidates are just that doc.
{
const f = [_]bson.Pair{.{ .key = "_id", .value = .{ .int32 = 3 } }};
var p = (try plan(gpa, &ix, &.{}, &f, &.{})).?;
defer p.deinit(gpa);
try testing.expectEqual(@as(usize, 1), p.key_len());
var ids: std.ArrayListUnmanaged([]const u8) = .empty;
defer ids.deinit(gpa);
try p.search(gpa, &ids);
try testing.expectEqual(@as(usize, 1), ids.items.len);
try testing.expectEqualStrings("d3", ids.items[0]);
}
// The same query as an int64 and a double finds the int32 entry.
{
const f = [_]bson.Pair{.{ .key = "_id", .value = .{ .int64 = 3 } }};
var p = (try plan(gpa, &ix, &.{}, &f, &.{})).?;
defer p.deinit(gpa);
var ids: std.ArrayListUnmanaged([]const u8) = .empty;
defer ids.deinit(gpa);
try p.search(gpa, &ids);
try testing.expectEqual(@as(usize, 1), ids.items.len);
try testing.expectEqualStrings("d3", ids.items[0]);
}
// A range on _id yields the band in index order.
{
const f = [_]bson.Pair{.{ .key = "_id", .value = .{ .doc = &.{.{ .key = "$gte", .value = .{ .int32 = 3 } }} } }};
var p = (try plan(gpa, &ix, &.{}, &f, &.{})).?;
defer p.deinit(gpa);
var ids: std.ArrayListUnmanaged([]const u8) = .empty;
defer ids.deinit(gpa);
try p.search(gpa, &ids);
try testing.expectEqual(@as(usize, 3), ids.items.len);
try testing.expectEqualStrings("d3", ids.items[0]);
try testing.expectEqualStrings("d5", ids.items[2]);
}
// sort({_id: 1}) with no filter: a full index scan that supplies the
// order — the plan the sort planner needs to stop materializing.
{
const sort = [_]query.SortKey{.{ .path = "_id", .descending = false }};
var p = (try plan(gpa, &ix, &.{}, &.{}, &sort)).?;
defer p.deinit(gpa);
try testing.expect(p.provides_sort and !p.backward);
var ids: std.ArrayListUnmanaged([]const u8) = .empty;
defer ids.deinit(gpa);
try p.search(gpa, &ids);
try testing.expectEqual(@as(usize, 5), ids.items.len);
try testing.expectEqualStrings("d1", ids.items[0]);
try testing.expectEqualStrings("d5", ids.items[4]);
}
// sort({_id: -1}): backward.
{
const sort = [_]query.SortKey{.{ .path = "_id", .descending = true }};
var p = (try plan(gpa, &ix, &.{}, &.{}, &sort)).?;
defer p.deinit(gpa);
try testing.expect(p.provides_sort and p.backward);
var ids: std.ArrayListUnmanaged([]const u8) = .empty;
defer ids.deinit(gpa);
try p.search(gpa, &ids);
try testing.expectEqualStrings("d5", ids.items[0]);
try testing.expectEqualStrings("d1", ids.items[4]);
}
// A filter naming no _id field leaves the _id index unusable.
{
const f = [_]bson.Pair{.{ .key = "v", .value = .{ .int32 = 1 } }};
try testing.expect((try plan(gpa, &ix, &.{}, &f, &.{})) == null);
}
}
test "planner picks eq run, ranges, and bails on sparse null" {
const gpa = testing.allocator;
var ix = try simple_index(gpa, &.{ "a", "b" }, false, false);
@@ -2484,7 +2427,7 @@ test "planner picks eq run, ranges, and bails on sparse null" {
.{ .key = "a", .value = .{ .int32 = 1 } },
.{ .key = "b", .value = .{ .int32 = 2 } },
};
var p = (try plan(gpa, &.{ix}, &f, &.{})).?;
var p = (try plan(gpa, null, &.{ix}, &f, &.{})).?;
defer p.deinit(gpa);
try testing.expectEqual(@as(usize, 2), p.key_len());
try testing.expect(p.lo == null and p.hi == null);
@@ -2495,7 +2438,7 @@ test "planner picks eq run, ranges, and bails on sparse null" {
.{ .key = "a", .value = .{ .int32 = 1 } },
.{ .key = "b", .value = .{ .doc = &.{.{ .key = "$gt", .value = .{ .int32 = 2 } }} } },
};
var p = (try plan(gpa, &.{ix}, &f, &.{})).?;
var p = (try plan(gpa, null, &.{ix}, &f, &.{})).?;
defer p.deinit(gpa);
try testing.expectEqual(@as(usize, 1), p.key_len());
try testing.expect(p.hi == null and p.lo != null and !p.lo_incl);
@@ -2503,14 +2446,14 @@ test "planner picks eq run, ranges, and bails on sparse null" {
// {a: 1} only → prefix run of 1.
{
const f = [_]bson.Pair{.{ .key = "a", .value = .{ .int32 = 1 } }};
var p = (try plan(gpa, &.{ix}, &f, &.{})).?;
var p = (try plan(gpa, null, &.{ix}, &f, &.{})).?;
defer p.deinit(gpa);
try testing.expectEqual(@as(usize, 1), p.key_len());
}
// Pure range on the first key → key_len 0 with a bound.
{
const f = [_]bson.Pair{.{ .key = "a", .value = .{ .doc = &.{.{ .key = "$gte", .value = .{ .int32 = 1 } }} } }};
var p = (try plan(gpa, &.{ix}, &f, &.{})).?;
var p = (try plan(gpa, null, &.{ix}, &f, &.{})).?;
defer p.deinit(gpa);
try testing.expectEqual(@as(usize, 0), p.key_len());
try testing.expect(p.lo != null and p.lo_incl);
@@ -2518,23 +2461,23 @@ test "planner picks eq run, ranges, and bails on sparse null" {
// Unusable filter → no plan.
{
const f = [_]bson.Pair{.{ .key = "a", .value = .{ .doc = &.{.{ .key = "$regex", .value = .{ .string = "^x" } }} } }};
try testing.expect((try plan(gpa, &.{ix}, &f, &.{})) == null);
try testing.expect((try plan(gpa, null, &.{ix}, &f, &.{})) == null);
const or_f = [_]bson.Pair{.{ .key = "$or", .value = .{ .array = &.{
.{ .doc = &.{.{ .key = "a", .value = .{ .int32 = 1 } }} },
} } }};
try testing.expect((try plan(gpa, &.{ix}, &or_f, &.{})) == null);
try testing.expect((try plan(gpa, null, &.{ix}, &or_f, &.{})) == null);
}
// Sparse index bails on a null component.
{
const f = [_]bson.Pair{.{ .key = "a", .value = .null }};
try testing.expect((try plan(gpa, &.{sp}, &f, &.{})) == null);
try testing.expect((try plan(gpa, null, &.{sp}, &f, &.{})) == null);
// Non-sparse is fine with null.
var p = (try plan(gpa, &.{ix}, &f, &.{})).?;
var p = (try plan(gpa, null, &.{ix}, &f, &.{})).?;
defer p.deinit(gpa);
try testing.expect(p.key_len() == 1);
// A null inside $in bails too.
const fin = [_]bson.Pair{.{ .key = "a", .value = .{ .doc = &.{.{ .key = "$in", .value = .{ .array = &.{ .{ .int32 = 1 }, .null } } }} } }};
try testing.expect((try plan(gpa, &.{sp}, &fin, &.{})) == null);
try testing.expect((try plan(gpa, null, &.{sp}, &fin, &.{})) == null);
}
// $in cartesian product is capped.
{
@@ -2545,6 +2488,6 @@ test "planner picks eq run, ranges, and bails on sparse null" {
.{ .key = "b", .value = .{ .doc = &.{.{ .key = "$in", .value = .{ .array = &members } }} } },
};
// 20 * 20 = 400 > 100 → fall back to a scan.
try testing.expect((try plan(gpa, &.{ix}, &f, &.{})) == null);
try testing.expect((try plan(gpa, null, &.{ix}, &f, &.{})) == null);
}
}

38
tests/e2e/results/phase3.txt Normal file
View File

@@ -0,0 +1,38 @@
# Phase 3 gate — mongo-lite vs MongoDB 8.3.7, 1g dataset / ~16k docs
# Ratio < 1.0 = mongo-lite faster. Reproduce: bash tests/e2e/compare-run.sh 1g 16k
# This run includes roadmap items 1 (B+tree) and 2 (ordered _id index).
# Compare: tests/e2e/results/phase1.txt (pre-tree baseline).
benchmark mongo-lite mongodb ratio
insertOne (sequential) ×200 0.19 ms 4.4 ms 0.0x
bulk insert throughput 816.5 MB/s 716.8 MB/s 1.1x
docs loaded 65,536 65,536 1.0x
createIndex({k: 1}) 50.7 ms 76.8 ms 0.7x
countDocuments({}) 1.5 ms 11.1 ms 0.1x
findOne({_id: <ObjectId>}) 0.57 ms 0.64 ms 0.9x
findOne({k: 500}) (indexed) 0.58 ms 1.5 ms 0.4x
find({p: {$gte,$lt}}).count() (scan) 22.1 ms 12.9 ms 1.7x
find({}).sort({_id:-1}).limit(20) 2.4 ms 2.2 ms 1.1x
find({}, {proj}).limit(1000) 3.7 ms 4.3 ms 0.9x
aggregate $group by k 9.1 ms 12.5 ms 0.7x
updateOne({_id}) ×50 0.16 ms 0.19 ms 0.8x
updateMany({k: 7}, {$inc}) 1.6 ms 6.4 ms 0.3x
deleteOne({_id}) + insertOne 0.62 ms 4.8 ms 0.1x
node client RSS 159 MB 156 MB 1.0x
server RSS 1973 MB 1334 MB
kill -9 reopen 0.8s 1.3s
db on disk 1025MB 91MB
# Item 2 (ordered _id index) deltas vs phase2:
# sort({_id:-1}).limit(20) 6.2 -> 2.4 ms (2.3x slower than mongod -> parity):
# the sort planner now scans the _id tree in order and stops
# at the page limit instead of materializing every candidate.
# integer/string _id findOne, updateOne, deleteOne no longer fall back to a
# full collection scan (verified separately: point lookups,
# $in, ranges, int64/int32 compare-equal equality all hit the
# tree; the docs-map serialization-guard fast path is gone).
#
# Remaining gaps and where they are addressed:
# db on disk 11x -> Phase 3 (block-compressed log)
# range-scan 1.7x -> Phase 4 (contiguous byte storage, not the matcher)
# server RSS 1.5x -> Phase 4 (per-document arena -> byte storage)