diff --git a/README.md b/README.md index 4fd2307..e30575b 100644 --- a/README.md +++ b/README.md @@ -190,14 +190,14 @@ With the ReleaseFast default build (MongoDB 8.3.7 on the same Mac): | createIndex({k: 1}) | 51 ms | 82 ms | **mongo-lite** | | countDocuments({}) | 1.5 ms | 13.8 ms | **mongo-lite ×9** | | findOne({_id}) | 0.57 ms | 0.67 ms | mongo-lite | -| findOne indexed | 0.57 ms | 1.8 ms | **mongo-lite ×3** | -| range-scan count | 20 ms | 13 ms | mongodb ×1.6 | -| sort + limit(20), on `_id` | 6.2 ms | 2.7 ms | mongodb ×2.3 | +| findOne indexed | 0.58 ms | 1.5 ms | **mongo-lite ×2.6** | +| range-scan count | 22 ms | 13 ms | mongodb ×1.7 | +| sort + limit(20), on `_id` | 2.4 ms | 2.2 ms | mongodb ×1.1 | | sort + limit(20), indexed field | 1.0 ms | — | — | | aggregate $group | 11.5 ms | 15.5 ms | **mongo-lite** | | updateOne({_id}) | 0.17 ms | 0.19 ms | mongo-lite | -| updateMany (65 docs) | 1.8 ms | 6.7 ms | **mongo-lite ×3.7** | -| deleteOne + insert | 0.50 ms | 5.0 ms | **mongo-lite ×10** | +| updateMany (65 docs) | 1.6 ms | 6.4 ms | **mongo-lite ×4** | +| deleteOne + insert | 0.62 ms | 4.8 ms | **mongo-lite ×8** | | server RSS | 2.0 GB | 1.5 GB | mongodb (×0.7) | | kill -9 → reopen | 0.8 s | 1.3 s | **mongo-lite** | | db on disk | 1.0 GB | 96 MB | mongodb (compressed) | @@ -206,15 +206,12 @@ The remaining losses are structural rather than incidental. Disk size is the big one: payloads are stored raw, so the log is 11x MongoDB's compressed files. RSS trails because every document carries its own arena. The range-scan gap is not the matcher — it is walking 65,536 documents -that each live in a separate allocation, one pointer chase apiece. And -`sort` on `_id` still materializes candidates because nothing ordered -covers `_id` yet; the same sort on an indexed field streams straight out -of the index at 1.0 ms. +that each live in a separate allocation, one pointer chase apiece. Reproduce the table with `bash tests/e2e/compare-run.sh 1g 16k`; the -pre-tree baseline is recorded in `tests/e2e/results/phase1.txt`, the run -above (with the B+tree, roadmap item 1) in -`tests/e2e/results/phase2.txt`. +pre-tree baseline is recorded in `tests/e2e/results/phase1.txt`, and the +runs with the B+tree and ordered `_id` index (roadmap items 1 and 2) in +`tests/e2e/results/phase2.txt` and `tests/e2e/results/phase3.txt`. ### What is left (highest impact first) @@ -226,15 +223,11 @@ traps in [ROADMAP.md](ROADMAP.md). highly compressible workloads massively; note Zig 0.16 ships zstd decompression only, and deflate would cap writes below the current insert rate. -2. **An ordered `_id` index** — `sort({_id: ...})` still materializes every - candidate, and integer `_id`s still scan. Both fall out of indexing the - encoded `_id`. The tree is in place, so an `_id` index update is a - leaf insert, not a tail memmove. -3. **Stop giving every document its own arena** — the source of both the +2. **Stop giving every document its own arena** — the source of both the RSS gap and the range-scan gap. Storing canonical BSON bytes in a per-collection slab and matching against them (parsing only the fields a filter names) makes scans contiguous instead of a pointer chase. -4. **Decompose the global lock** — one reader/writer lock covers the whole +3. **Decompose the global lock** — one reader/writer lock covers the whole engine and is held across fsync, compaction and reply construction. Per-collection locks plus cross-connection group commit are the path to using more than one core on writes. @@ -257,8 +250,16 @@ Done so far, with the measurement that drove each: records, no rebalancing on delete, and bulk bottom-up packing. Entry insertion and removal are a descent plus a leaf-local edit instead of a tail memmove, so writes into an already-built index stopped being - quadratic. `updateMany` 17.3 → 1.8 ms (2.8x slower than MongoDB → 3.7x + quadratic. `updateMany` 17.3 → 1.6 ms (2.8x slower than MongoDB → 4x faster); `createIndex` 62 → 51 ms. +- **An ordered `_id` index** (roadmap item 2): every collection carries an + implicit `_id_` index (kept out of the secondary list, so the listing, + drop and log-format surfaces are unchanged; rebuilt after replay like + the secondaries). Its encoded keys are canonical, so the old + serialization-guarded docs-map fast path is gone and integer/string + `_id` point lookups, `$in` and ranges hit the tree instead of a full + scan. `sort({_id: ...})` is now an index-ordered scan with an early stop: + `sort+limit(20)` 6.2 → 2.4 ms (parity with MongoDB). - **Entry removal is a binary search**, not a scan of the whole index. `updateMany` 15.4 → 5.5 ms. - **Top-k sort selection** and an allocation-free decorate pass, plus diff --git a/ROADMAP.md b/ROADMAP.md index 18e8faa..f74072b 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,9 +1,10 @@ # Remaining performance work -Status: **item 1 (B+tree over the encoded keys) is done** — landed and -verified in `tests/e2e/results/phase2.txt` (updateMany 17.3 → 1.8 ms, -createIndex 62 → 51 ms). Its dependents (items 2 and 4) now stand on a -tree instead of a sorted array. Five items below, in dependency order. +Status: **items 1 (B+tree over the encoded keys) and 2 (ordered `_id` index) +are done** — landed and verified in `tests/e2e/results/phase2.txt` and +`phase3.txt` (updateMany 17.3 → 1.6 ms, createIndex 62 → 51 ms, `_id` +sort+limit 6.2 → 2.4 ms). Their dependents (items 4) now stand on a tree +instead of a sorted array. Items below, in dependency order. Each is sized to be landed and verified on its own; the ordering constraints between them are the load-bearing part, so read those before picking one up. @@ -101,7 +102,26 @@ differential. Both already exist and both are mutation-checked. --- -## 2. Ordered `_id` index — **depends on 1** +## 2. Ordered `_id` index — DONE + +Landed as an implicit `_id_` index on every `Collection` (a normal +`index.Index` with keys `[_id: 1]`, kept out of the secondary `indexes` +list so `listIndexes`/`dropIndexes`/`createIndex` and the log format are +unchanged — no `index_create` record, no double listing). Maintained in +`upsert` (through the same reserve-then-insert protocol as the +secondaries) and `evict_doc`; rebuilt after replay by `build_all_indexes` +alongside the secondaries. `index.plan` now takes it as a separate +argument, so `{_id: ...}` equality, `$in` and ranges use the tree (the +old serialization-guarded docs-map fast path — `plan_id`, +`value_fast_path_safe` and friends — is deleted), and `sort({_id: ...})` +becomes an index-ordered full scan with an early stop. A full `_id` scan +cannot miss a document: every doc has an `_id` and the index is not +sparse, and its keys are canonical (`bson.encode_key` gives int32 1, +int64 1 and double 1.0 identical bytes). + +Recorded deltas vs `tests/e2e/results/phase2.txt`: `sort({_id:-1}).limit(20)` +6.2 → 2.4 ms (2.3x slower than MongoDB → parity). Integer/string `_id` +point lookups, `$in` and ranges no longer full-scan. **Why.** `sort({_id: ...})` still materializes every candidate, and integer `_id`s still fall back to a full collection scan on every `findOne`, @@ -115,8 +135,8 @@ requires. Then delete `value_fast_path_safe` and friends (`src/index.zig`). They exist only because `serialize_value` gives `int32 1`, `int64 1` and `double 1.0` -different bytes despite comparing equal. `bson.encode_key` already gives them -identical bytes, so the guard is obsolete. +different bytes despite comparing equal. `bson.encode_key` already gives +them identical bytes, so the guard is obsolete. **Why it depends on item 1.** This index updates on *every* insert. Against a sorted array that is a tail memmove each time — roughly 51 GB of memmove over @@ -128,7 +148,9 @@ only" caveat. **Watch.** A real `_id_` index may start appearing in `listIndexes` and writing an `index_create` record to the log. `e2e3.js` asserts on index -listings — check it before assuming this is invisible. +listings — check it before assuming this is invisible. (This landed without +either: the index stays out of the secondary list, so the listing, drop and +log surfaces are untouched; verified with `e2e3.js` unchanged.) --- diff --git a/src/commands.zig b/src/commands.zig index c41fe2c..ac5924a 100644 --- a/src/commands.zig +++ b/src/commands.zig @@ -644,31 +644,10 @@ fn scan_sorted( const filter_doc = bson.Document{ .arena = undefined, .pairs = filter }; var n: usize = 0; - // _id_ fast path: the docs map is the _id index. Skipped when the - // queried value's compare-equivalence class is serialization-ambiguous - // (see index.plan_id). - if (try index.plan_id(ctx.gpa, filter)) |id_plan| { - var plan = id_plan; - defer plan.deinit(ctx.gpa); - // One scratch key, rebuilt per value: a key is never needed past its - // own lookup. - var key: std.ArrayListUnmanaged(u8) = .empty; - defer key.deinit(ctx.gpa); - for (plan.values) |v| { - key.clearRetainingCapacity(); - try bson.write_serialized_value(v, ctx.gpa, &key); - const doc = coll.docs.get(key.items) orelse continue; - if (!try query.matches(ctx.gpa, &filter_doc, doc)) continue; - if (out) |list| try list.append(ctx.gpa, doc); - n += 1; - if (lim != 0 and n >= lim) break; - } - return n; - } - - // Secondary-index plan: candidates in index order, re-filtered. The - // returned ids alias the docs map keys, valid under the read lock. - if (try index.plan(ctx.gpa, coll.indexes.items, filter, sort)) |p| { + // Index plan (the implicit _id_ index first, then the secondaries): + // candidates in index order, re-filtered. The returned ids alias the + // docs map keys, valid under the read lock. + if (try index.plan(ctx.gpa, &coll.id_index, coll.indexes.items, filter, sort)) |p| { var plan = p; defer plan.deinit(ctx.gpa); var ids: std.ArrayListUnmanaged([]const u8) = .empty; diff --git a/src/db.zig b/src/db.zig index dc7ee55..8371011 100644 --- a/src/db.zig +++ b/src/db.zig @@ -12,10 +12,23 @@ const index = @import("index.zig"); pub const Collection = struct { docs: std.StringHashMapUnmanaged(*bson.Document), + /// Secondary indexes (persisted through the log). indexes: std.ArrayListUnmanaged(index.Index), + /// The implicit _id_ index: every document has an _id and it is not + /// sparse, so entry count equals document count and a full scan of it + /// cannot miss a document — which is what the sort planner's full-scan + /// plan relies on. Kept out of `indexes` so the listing/drop commands + /// and the log format are unchanged (it is rebuilt on open like + /// everything else). `bson.encode_key` keys are canonical, so it also + /// replaces the old serialization-guarded docs-map fast path for + /// integer/string/etc. _id lookups. + id_index: index.Index, - fn init() Collection { - return .{ .docs = .empty, .indexes = .empty }; + fn init(gpa: std.mem.Allocator) !Collection { + var self: Collection = .{ .docs = .empty, .indexes = .empty, .id_index = undefined }; + const keys = [_]index.IndexKey{.{ .path = "_id", .descending = false }}; + self.id_index = try index.Index.init(gpa, "_id_", &keys, false, false, null); + return self; } /// The secondary index with this name, or null. The single by-name @@ -108,6 +121,7 @@ pub const Engine = struct { // Dropping a collection turns all of its records into garbage. self.live_docs -= coll.docs.count(); self.dead_docs += coll.docs.count(); + coll.id_index.deinit(self.gpa); for (coll.indexes.items) |*ix| ix.deinit(self.gpa); coll.indexes.deinit(self.gpa); var doc_it = coll.docs.iterator(); @@ -139,6 +153,7 @@ pub const Engine = struct { /// regenerating them from it, which is far cheaper than scanning. fn evict_doc(self: *Engine, coll: *Collection, id_key: []const u8) void { const old = coll.docs.fetchRemove(id_key) orelse return; + coll.id_index.remove_doc(self.gpa, old.value, old.key); for (coll.indexes.items) |*ix| ix.remove_doc(self.gpa, old.value, old.key); old.value.*.deinit(); self.gpa.destroy(old.value); @@ -241,6 +256,15 @@ pub const Engine = struct { return err; }; } + { + // The implicit _id_ index, through the same protocol: reserved + // before the log append, inserted infallibly after it. + var built = try coll.id_index.build_entries(self.gpa, owned, id_key); + built_list.append(self.gpa, .{ .built = built, .ix = &coll.id_index }) catch |err| { + built.deinit(self.gpa); + return err; + }; + } // 2. The _id check, mirroring the pre-index behavior. if (mode == .insert and coll.docs.contains(id_key)) return error.DuplicateKey; @@ -494,7 +518,9 @@ pub const Engine = struct { if (db.collections.getPtr(coll_name)) |coll| return coll; const coll_key = try self.gpa.dupe(u8, coll_name); errdefer self.gpa.free(coll_key); - try db.collections.put(self.gpa, coll_key, Collection.init()); + var new_coll = try Collection.init(self.gpa); + errdefer new_coll.id_index.deinit(self.gpa); + try db.collections.put(self.gpa, coll_key, new_coll); return db.collections.getPtr(coll_name) orelse unreachable; } @@ -624,26 +650,37 @@ pub const Engine = struct { var coll_it = db_entry.value_ptr.collections.iterator(); while (coll_it.next()) |coll_entry| { for (coll_entry.value_ptr.indexes.items) |*ix| { - if (ix.count() > 0) continue; // defensive - var doc_it = coll_entry.value_ptr.docs.iterator(); - while (doc_it.next()) |doc_entry| { - ix.append_doc_entries(self.gpa, doc_entry.value_ptr.*, doc_entry.key_ptr.*) catch |err| switch (err) { - error.ParallelArrays => { - std.debug.print("mongo-lite: WARNING: index '{s}' cannot index an existing document; entry skipped\n", .{ix.name}); - continue; - }, - else => return err, - }; - } - // Tolerated, not enforced: the database must always open. - if (try ix.finish_bulk(self.gpa, false)) { - std.debug.print("mongo-lite: WARNING: unique index '{s}' has duplicate keys in existing data; duplicates not enforced for existing documents\n", .{ix.name}); - } + try self.rebuild_index(coll_entry.value_ptr, ix); } + try self.rebuild_index(coll_entry.value_ptr, &coll_entry.value_ptr.id_index); } } } + /// Rebuild one index from the live documents. Runs after replay, so it + /// is order-independent; indexes already holding entries (maintained + /// live) are skipped defensively. A duplicate under a unique index logs + /// a loud warning and keeps the index (still correct as a candidate + /// generator; future writes are still enforced) — the database always + /// opens, leaving dropIndexes as an in-band recovery path. + fn rebuild_index(self: *Engine, coll: *Collection, ix: *index.Index) !void { + if (ix.count() > 0) return; // defensive + var doc_it = coll.docs.iterator(); + while (doc_it.next()) |doc_entry| { + ix.append_doc_entries(self.gpa, doc_entry.value_ptr.*, doc_entry.key_ptr.*) catch |err| switch (err) { + error.ParallelArrays => { + std.debug.print("mongo-lite: WARNING: index '{s}' cannot index an existing document; entry skipped\n", .{ix.name}); + continue; + }, + else => return err, + }; + } + // Tolerated, not enforced: the database must always open. + if (try ix.finish_bulk(self.gpa, false)) { + std.debug.print("mongo-lite: WARNING: unique index '{s}' has duplicate keys in existing data; duplicates not enforced for existing documents\n", .{ix.name}); + } + } + /// Register an (empty) index from a persisted spec document. A repeated /// create record for the same name is an idempotent no-op. fn register_index_from_spec(self: *Engine, coll: *Collection, spec_doc: *const bson.Document) !void { @@ -721,6 +758,8 @@ fn apply_record(ctx: *anyopaque, record: storage.Record, doc: *bson.Document) an self.live_docs += 1; key_owned = true; stored = true; + // The _id_ entry is added after replay, in build_all_indexes, + // together with the secondary indexes. }, storage.record_type_delete => self.evict_doc(coll, id_key), else => {}, diff --git a/src/index.zig b/src/index.zig index 3a4f56f..fb86cac 100644 --- a/src/index.zig +++ b/src/index.zig @@ -1523,13 +1523,32 @@ pub const Plan = struct { /// documents: the one covering the longest leading run of equality/$in /// predicates, optionally with a range on the next key. Returns null when /// nothing usable remains — the caller scans. -pub fn plan(gpa: std.mem.Allocator, indexes: []const Index, filter: []const bson.Pair, sort: []const query.SortKey) !?Plan { - if (indexes.len == 0) return null; +/// +/// `id_ix` is the collection's implicit `_id_` index (kept separate from +/// the secondary `indexes` list so the listing/drop commands stay +/// unchanged). Every document has an `_id` and the index is not sparse, so +/// a full scan of it cannot miss a document — which is what the sort +/// planner's full-scan plan relies on. The encoded keys are canonical, so +/// `_id` equality on compare-equal values (int32 1, int64 1, double 1.0) +/// finds the same entries. +pub fn plan( + gpa: std.mem.Allocator, + id_ix: ?*const Index, + indexes: []const Index, + filter: []const bson.Pair, + sort: []const query.SortKey, +) !?Plan { + if (id_ix == null and indexes.len == 0) return null; var clauses: std.ArrayListUnmanaged(Clause) = .empty; defer clauses.deinit(gpa); try flatten_clauses(gpa, filter, &clauses); var best: ?Plan = null; + if (id_ix) |idx| { + if (try evaluate_index(gpa, idx, clauses.items, sort)) |cand| { + best = cand; + } + } for (indexes) |*ix| { var cand = (try evaluate_index(gpa, ix, clauses.items, sort)) orelse continue; if (best) |b| { @@ -1682,112 +1701,6 @@ fn evaluate_index(gpa: std.mem.Allocator, ix: *const Index, clauses: []const Cla return pl; } -// --------------------------------------------------------------------------- -// _id fast path -// --------------------------------------------------------------------------- - -pub const IdPlan = struct { - /// Values to look up in the docs map. A single $eq value is a heap copy - /// owned by this plan; a $in list aliases the filter's array. Freed by - /// deinit. - values: []const bson.Value, - owned: bool, - - pub fn deinit(self: *IdPlan, gpa: std.mem.Allocator) void { - if (self.owned) gpa.free(self.values); - } -}; - -/// Plan for the implicit _id_ index (the docs map). Only applies to a -/// top-level _id equality/$eq/$in clause, and only when the queried value's -/// compare-equivalence class is serialization-canonical. bson.compare calls -/// int32 1, int64 1 and double 1.0 equal (and string/symbol/code "x" equal, -/// and nested variants), but serialize_value produces different map keys — -/// a hash lookup would then miss documents a scan would match. -/// -/// The returned plan must be deinit'd: the single-value case is heap-copied -/// so the slice never points into a temporary (ReleaseFast reuses the stack, -/// which turned a dangling anonymous-list pointer into garbage). -pub fn plan_id(gpa: std.mem.Allocator, filter: []const bson.Pair) error{OutOfMemory}!?IdPlan { - // The first usable _id clause wins; no flattening buffer is needed - // because nothing is compared across clauses. $and members are searched - // like top-level pairs, every other operator skipped — same rule as - // flatten_clauses, and safe for the same reason (the full filter is - // re-applied to every candidate). - for (filter) |p| { - if (p.key.len > 0 and p.key[0] == '$') { - if (!std.mem.eql(u8, p.key, "$and")) continue; - const members = switch (p.value) { - .array => |a| a, - else => continue, - }; - for (members) |m| { - const mp = switch (m) { - .doc => |d| d, - else => continue, - }; - if (try plan_id(gpa, mp)) |found| return found; - } - continue; - } - if (!std.mem.eql(u8, p.key, "_id")) continue; - if (try id_lookup_values(gpa, p.value)) |lookup| return .{ .values = lookup.values, .owned = lookup.owned }; - } - return null; -} - -const IdLookup = struct { - values: []const bson.Value, - owned: bool, -}; - -/// The map-lookup values for one _id clause, or null when it is not a pure -/// equality/$in of fast-path-safe values. A range is unusable here: the docs -/// map is a hash, not an ordered structure. -fn id_lookup_values(gpa: std.mem.Allocator, v: bson.Value) error{OutOfMemory}!?IdLookup { - var info = CompInfo{}; - analyze_clause(v, &info); - if (info.lo != null or info.hi != null) return null; - if (info.eq) |e| { - if (!value_fast_path_safe(e)) return null; - // Heap-copy the single value: a pointer to a stack or anonymous - // temporary would dangle once this frame returns. - const buf = try gpa.alloc(bson.Value, 1); - buf[0] = e; - return .{ .values = buf[0..1], .owned = true }; - } - if (info.in_values) |list| { - for (list) |m| { - if (!value_fast_path_safe(m)) return null; - } - return .{ .values = list, .owned = false }; - } - return null; -} - -/// Whether serialize_value is injective on the value's compare-equivalence -/// class. False for numbers (int32 1 / int64 1 / double 1.0), string/symbol/ -/// code (compared equal, serialized with different type tags), opaque -/// values (compared by payload only), and anything containing those. -fn value_fast_path_safe(v: bson.Value) bool { - return switch (v) { - .double, .int32, .int64, .string, .symbol, .code, .opaque_val => false, - .doc => |pairs| blk: { - for (pairs) |p| { - if (!value_fast_path_safe(p.value)) break :blk false; - } - break :blk true; - }, - .array => |items| blk: { - for (items) |it| { - if (!value_fast_path_safe(it)) break :blk false; - } - break :blk true; - }, - else => true, - }; -} - // --------------------------------------------------------------------------- // Tests // --------------------------------------------------------------------------- @@ -2301,63 +2214,6 @@ test "lookup_range matches a brute-force filter over random data" { } } -test "id fast path guards and $in" { - // plan_id may heap-copy the single value; run it through the allocator - // and free. The helper asserts on whether a plan was produced. - const gpa = testing.allocator; - const plans = struct { - fn has(pairs: []const bson.Pair) !bool { - var p = try plan_id(gpa, pairs); - defer if (p) |*pl| pl.deinit(gpa); - return p != null; - } - }; - // Numbers never use the fast path (compare-equal but serialize-different). - try testing.expect(!try plans.has(&.{.{ .key = "_id", .value = .{ .int32 = 1 } }})); - // Strings are skipped too: a stored symbol/code _id compare-equals a - // string query but serializes differently. - try testing.expect(!try plans.has(&.{.{ .key = "_id", .value = .{ .string = "x" } }})); - // Truly canonical values (bool, ObjectId) do use it. - try testing.expect(try plans.has(&.{.{ .key = "_id", .value = .{ .bool = true } }})); - const oid = [_]u8{ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 }; - try testing.expect(try plans.has(&.{.{ .key = "_id", .value = .{ .object_id = oid } }})); - // $in with a number member is skipped. - const mixed = [_]bson.Pair{.{ .key = "_id", .value = .{ .doc = &.{ - .{ .key = "$in", .value = .{ .array = &.{ .{ .bool = true }, .{ .int32 = 1 } } } }, - } } }}; - try testing.expect(!try plans.has(&mixed)); - const safe_in = [_]bson.Pair{.{ .key = "_id", .value = .{ .doc = &.{ - .{ .key = "$in", .value = .{ .array = &.{ .{ .bool = true }, .{ .bool = false } } } }, - } } }}; - try testing.expect(try plans.has(&safe_in)); - // $and members count as top-level. - const and_f = [_]bson.Pair{.{ .key = "$and", .value = .{ .array = &.{ - .{ .doc = &.{.{ .key = "_id", .value = .{ .bool = true } }} }, - } } }}; - try testing.expect(try plans.has(&and_f)); - // $or is not usable for the fast path. - const or_f = [_]bson.Pair{.{ .key = "$or", .value = .{ .array = &.{ - .{ .doc = &.{.{ .key = "_id", .value = .{ .bool = true } }} }, - } } }}; - try testing.expect(!try plans.has(&or_f)); - // A doc containing a number is not fast-path safe. - const doc_id = [_]bson.Pair{.{ .key = "_id", .value = .{ .doc = &.{.{ .key = "n", .value = .{ .int32 = 1 } }} } }}; - try testing.expect(!try plans.has(&doc_id)); - // A doc containing a string is unsafe too (string/symbol/code class). - const doc_str = [_]bson.Pair{.{ .key = "_id", .value = .{ .doc = &.{.{ .key = "n", .value = .{ .string = "x" } }} } }}; - try testing.expect(!try plans.has(&doc_str)); - // A doc of canonical values is safe. - const doc_safe = [_]bson.Pair{.{ .key = "_id", .value = .{ .doc = &.{.{ .key = "n", .value = .{ .bool = true } }} } }}; - try testing.expect(try plans.has(&doc_safe)); - // The single-value plan owns a usable copy: the ObjectId must read back - // exactly (this is the ReleaseFast regression the fix guards — a dangling - // pointer read back as garbage). - var p = try plan_id(gpa, &.{.{ .key = "_id", .value = .{ .object_id = oid } }}); - defer if (p) |*pl| pl.deinit(gpa); - try testing.expect(p != null); - try testing.expectEqualSlices(u8, &oid, &p.?.values[0].object_id); -} - test "TTL spec round-trips through write_spec and compares in spec_equal" { const gpa = testing.allocator; const spec = doc_of(&.{ @@ -2471,6 +2327,93 @@ test "TTL spec rejects compound keys and bad expireAfterSeconds" { try testing.expectEqualStrings("expireAt_1", ix.name); } +test "the _id index plan covers equality, ranges and _id sort order" { + // The implicit _id_ index is a normal Index (keys = [_id: 1]) passed to + // plan separately from the secondaries. Its encoded keys are canonical, + // so compare-equal numbers (int32/int64/double) find the same entries, + // and a full scan of it is the sort planner's order supply for + // sort({_id: ...}). + const gpa = testing.allocator; + var ix = try simple_index(gpa, &.{"_id"}, false, false); + defer ix.deinit(gpa); + for (0..5) |i| { + const d = doc_of(&.{ + .{ .key = "_id", .value = .{ .int32 = @intCast(i + 1) } }, + .{ .key = "v", .value = .{ .int32 = @intCast(i) } }, + }); + const id = try std.fmt.allocPrint(gpa, "d{d}", .{i + 1}); + defer gpa.free(id); + _ = try ix.add_doc(gpa, &d, id, false); + } + + // {_id: 3} → an equality plan whose candidates are just that doc. + { + const f = [_]bson.Pair{.{ .key = "_id", .value = .{ .int32 = 3 } }}; + var p = (try plan(gpa, &ix, &.{}, &f, &.{})).?; + defer p.deinit(gpa); + try testing.expectEqual(@as(usize, 1), p.key_len()); + var ids: std.ArrayListUnmanaged([]const u8) = .empty; + defer ids.deinit(gpa); + try p.search(gpa, &ids); + try testing.expectEqual(@as(usize, 1), ids.items.len); + try testing.expectEqualStrings("d3", ids.items[0]); + } + // The same query as an int64 and a double finds the int32 entry. + { + const f = [_]bson.Pair{.{ .key = "_id", .value = .{ .int64 = 3 } }}; + var p = (try plan(gpa, &ix, &.{}, &f, &.{})).?; + defer p.deinit(gpa); + var ids: std.ArrayListUnmanaged([]const u8) = .empty; + defer ids.deinit(gpa); + try p.search(gpa, &ids); + try testing.expectEqual(@as(usize, 1), ids.items.len); + try testing.expectEqualStrings("d3", ids.items[0]); + } + // A range on _id yields the band in index order. + { + const f = [_]bson.Pair{.{ .key = "_id", .value = .{ .doc = &.{.{ .key = "$gte", .value = .{ .int32 = 3 } }} } }}; + var p = (try plan(gpa, &ix, &.{}, &f, &.{})).?; + defer p.deinit(gpa); + var ids: std.ArrayListUnmanaged([]const u8) = .empty; + defer ids.deinit(gpa); + try p.search(gpa, &ids); + try testing.expectEqual(@as(usize, 3), ids.items.len); + try testing.expectEqualStrings("d3", ids.items[0]); + try testing.expectEqualStrings("d5", ids.items[2]); + } + // sort({_id: 1}) with no filter: a full index scan that supplies the + // order — the plan the sort planner needs to stop materializing. + { + const sort = [_]query.SortKey{.{ .path = "_id", .descending = false }}; + var p = (try plan(gpa, &ix, &.{}, &.{}, &sort)).?; + defer p.deinit(gpa); + try testing.expect(p.provides_sort and !p.backward); + var ids: std.ArrayListUnmanaged([]const u8) = .empty; + defer ids.deinit(gpa); + try p.search(gpa, &ids); + try testing.expectEqual(@as(usize, 5), ids.items.len); + try testing.expectEqualStrings("d1", ids.items[0]); + try testing.expectEqualStrings("d5", ids.items[4]); + } + // sort({_id: -1}): backward. + { + const sort = [_]query.SortKey{.{ .path = "_id", .descending = true }}; + var p = (try plan(gpa, &ix, &.{}, &.{}, &sort)).?; + defer p.deinit(gpa); + try testing.expect(p.provides_sort and p.backward); + var ids: std.ArrayListUnmanaged([]const u8) = .empty; + defer ids.deinit(gpa); + try p.search(gpa, &ids); + try testing.expectEqualStrings("d5", ids.items[0]); + try testing.expectEqualStrings("d1", ids.items[4]); + } + // A filter naming no _id field leaves the _id index unusable. + { + const f = [_]bson.Pair{.{ .key = "v", .value = .{ .int32 = 1 } }}; + try testing.expect((try plan(gpa, &ix, &.{}, &f, &.{})) == null); + } +} + test "planner picks eq run, ranges, and bails on sparse null" { const gpa = testing.allocator; var ix = try simple_index(gpa, &.{ "a", "b" }, false, false); @@ -2484,7 +2427,7 @@ test "planner picks eq run, ranges, and bails on sparse null" { .{ .key = "a", .value = .{ .int32 = 1 } }, .{ .key = "b", .value = .{ .int32 = 2 } }, }; - var p = (try plan(gpa, &.{ix}, &f, &.{})).?; + var p = (try plan(gpa, null, &.{ix}, &f, &.{})).?; defer p.deinit(gpa); try testing.expectEqual(@as(usize, 2), p.key_len()); try testing.expect(p.lo == null and p.hi == null); @@ -2495,7 +2438,7 @@ test "planner picks eq run, ranges, and bails on sparse null" { .{ .key = "a", .value = .{ .int32 = 1 } }, .{ .key = "b", .value = .{ .doc = &.{.{ .key = "$gt", .value = .{ .int32 = 2 } }} } }, }; - var p = (try plan(gpa, &.{ix}, &f, &.{})).?; + var p = (try plan(gpa, null, &.{ix}, &f, &.{})).?; defer p.deinit(gpa); try testing.expectEqual(@as(usize, 1), p.key_len()); try testing.expect(p.hi == null and p.lo != null and !p.lo_incl); @@ -2503,14 +2446,14 @@ test "planner picks eq run, ranges, and bails on sparse null" { // {a: 1} only → prefix run of 1. { const f = [_]bson.Pair{.{ .key = "a", .value = .{ .int32 = 1 } }}; - var p = (try plan(gpa, &.{ix}, &f, &.{})).?; + var p = (try plan(gpa, null, &.{ix}, &f, &.{})).?; defer p.deinit(gpa); try testing.expectEqual(@as(usize, 1), p.key_len()); } // Pure range on the first key → key_len 0 with a bound. { const f = [_]bson.Pair{.{ .key = "a", .value = .{ .doc = &.{.{ .key = "$gte", .value = .{ .int32 = 1 } }} } }}; - var p = (try plan(gpa, &.{ix}, &f, &.{})).?; + var p = (try plan(gpa, null, &.{ix}, &f, &.{})).?; defer p.deinit(gpa); try testing.expectEqual(@as(usize, 0), p.key_len()); try testing.expect(p.lo != null and p.lo_incl); @@ -2518,23 +2461,23 @@ test "planner picks eq run, ranges, and bails on sparse null" { // Unusable filter → no plan. { const f = [_]bson.Pair{.{ .key = "a", .value = .{ .doc = &.{.{ .key = "$regex", .value = .{ .string = "^x" } }} } }}; - try testing.expect((try plan(gpa, &.{ix}, &f, &.{})) == null); + try testing.expect((try plan(gpa, null, &.{ix}, &f, &.{})) == null); const or_f = [_]bson.Pair{.{ .key = "$or", .value = .{ .array = &.{ .{ .doc = &.{.{ .key = "a", .value = .{ .int32 = 1 } }} }, } } }}; - try testing.expect((try plan(gpa, &.{ix}, &or_f, &.{})) == null); + try testing.expect((try plan(gpa, null, &.{ix}, &or_f, &.{})) == null); } // Sparse index bails on a null component. { const f = [_]bson.Pair{.{ .key = "a", .value = .null }}; - try testing.expect((try plan(gpa, &.{sp}, &f, &.{})) == null); + try testing.expect((try plan(gpa, null, &.{sp}, &f, &.{})) == null); // Non-sparse is fine with null. - var p = (try plan(gpa, &.{ix}, &f, &.{})).?; + var p = (try plan(gpa, null, &.{ix}, &f, &.{})).?; defer p.deinit(gpa); try testing.expect(p.key_len() == 1); // A null inside $in bails too. const fin = [_]bson.Pair{.{ .key = "a", .value = .{ .doc = &.{.{ .key = "$in", .value = .{ .array = &.{ .{ .int32 = 1 }, .null } } }} } }}; - try testing.expect((try plan(gpa, &.{sp}, &fin, &.{})) == null); + try testing.expect((try plan(gpa, null, &.{sp}, &fin, &.{})) == null); } // $in cartesian product is capped. { @@ -2545,6 +2488,6 @@ test "planner picks eq run, ranges, and bails on sparse null" { .{ .key = "b", .value = .{ .doc = &.{.{ .key = "$in", .value = .{ .array = &members } }} } }, }; // 20 * 20 = 400 > 100 → fall back to a scan. - try testing.expect((try plan(gpa, &.{ix}, &f, &.{})) == null); + try testing.expect((try plan(gpa, null, &.{ix}, &f, &.{})) == null); } } diff --git a/tests/e2e/results/phase3.txt b/tests/e2e/results/phase3.txt new file mode 100644 index 0000000..a1813a7 --- /dev/null +++ b/tests/e2e/results/phase3.txt @@ -0,0 +1,38 @@ +# Phase 3 gate — mongo-lite vs MongoDB 8.3.7, 1g dataset / ~16k docs +# Ratio < 1.0 = mongo-lite faster. Reproduce: bash tests/e2e/compare-run.sh 1g 16k +# This run includes roadmap items 1 (B+tree) and 2 (ordered _id index). +# Compare: tests/e2e/results/phase1.txt (pre-tree baseline). + +benchmark mongo-lite mongodb ratio +insertOne (sequential) ×200 0.19 ms 4.4 ms 0.0x +bulk insert throughput 816.5 MB/s 716.8 MB/s 1.1x +docs loaded 65,536 65,536 1.0x +createIndex({k: 1}) 50.7 ms 76.8 ms 0.7x +countDocuments({}) 1.5 ms 11.1 ms 0.1x +findOne({_id: }) 0.57 ms 0.64 ms 0.9x +findOne({k: 500}) (indexed) 0.58 ms 1.5 ms 0.4x +find({p: {$gte,$lt}}).count() (scan) 22.1 ms 12.9 ms 1.7x +find({}).sort({_id:-1}).limit(20) 2.4 ms 2.2 ms 1.1x +find({}, {proj}).limit(1000) 3.7 ms 4.3 ms 0.9x +aggregate $group by k 9.1 ms 12.5 ms 0.7x +updateOne({_id}) ×50 0.16 ms 0.19 ms 0.8x +updateMany({k: 7}, {$inc}) 1.6 ms 6.4 ms 0.3x +deleteOne({_id}) + insertOne 0.62 ms 4.8 ms 0.1x +node client RSS 159 MB 156 MB 1.0x +server RSS 1973 MB 1334 MB +kill -9 reopen 0.8s 1.3s +db on disk 1025MB 91MB + +# Item 2 (ordered _id index) deltas vs phase2: +# sort({_id:-1}).limit(20) 6.2 -> 2.4 ms (2.3x slower than mongod -> parity): +# the sort planner now scans the _id tree in order and stops +# at the page limit instead of materializing every candidate. +# integer/string _id findOne, updateOne, deleteOne no longer fall back to a +# full collection scan (verified separately: point lookups, +# $in, ranges, int64/int32 compare-equal equality all hit the +# tree; the docs-map serialization-guard fast path is gone). +# +# Remaining gaps and where they are addressed: +# db on disk 11x -> Phase 3 (block-compressed log) +# range-scan 1.7x -> Phase 4 (contiguous byte storage, not the matcher) +# server RSS 1.5x -> Phase 4 (per-document arena -> byte storage)