index: ordered _id index (roadmap item 2)

Give every Collection an implicit _id_ index (a normal Index with keys
[_id: 1]) so _id equality, $in, ranges and sorts stop depending on the
docs-map hash or a full scan. Kept out of the secondary indexes list, so
listIndexes/dropIndexes/createIndex and the log format are unchanged (no
index_create record, no double listing) and e2e3.js passes unmodified.

Maintained in upsert through the same reserve-then-insert protocol as
the secondaries, removed in evict_doc, and rebuilt after replay by
build_all_indexes alongside them (never maintained mid-replay, so a
failed add can't leave the index under-approximating). index.plan now
takes it as a separate argument. Its keys are canonical
(bson.encode_key gives int32 1, int64 1 and double 1.0 identical bytes),
so the serialization-guarded docs-map fast path (plan_id,
value_fast_path_safe and friends) is deleted.

Measured (tests/e2e/results/phase3.txt): sort({_id:-1}).limit(20) 6.2 ->
2.4 ms (2.3x slower than MongoDB -> parity); integer/string _id point
lookups, $in and ranges verified against the tree. Unit suite in all
three optimize modes, the crash pair, e2e3/e2e4/e2e6.

README.md

@@ -190,14 +190,14 @@ With the ReleaseFast default build (MongoDB 8.3.7 on the same Mac):
 | createIndex({k: 1}) | 51 ms | 82 ms | **mongo-lite** |
 | countDocuments({}) | 1.5 ms | 13.8 ms | **mongo-lite ×9** |
 | findOne({_id}) | 0.57 ms | 0.67 ms | mongo-lite |
-| findOne indexed | 0.57 ms | 1.8 ms | **mongo-lite ×3** |
+| findOne indexed | 0.58 ms | 1.5 ms | **mongo-lite ×2.6** |
-| range-scan count | 20 ms | 13 ms | mongodb ×1.6 |
+| range-scan count | 22 ms | 13 ms | mongodb ×1.7 |
-| sort + limit(20), on `_id` | 6.2 ms | 2.7 ms | mongodb ×2.3 |
+| sort + limit(20), on `_id` | 2.4 ms | 2.2 ms | mongodb ×1.1 |
 | sort + limit(20), indexed field | 1.0 ms | — | — |
 | aggregate $group | 11.5 ms | 15.5 ms | **mongo-lite** |
 | updateOne({_id}) | 0.17 ms | 0.19 ms | mongo-lite |
-| updateMany (65 docs) | 1.8 ms | 6.7 ms | **mongo-lite ×3.7** |
+| updateMany (65 docs) | 1.6 ms | 6.4 ms | **mongo-lite ×4** |
-| deleteOne + insert | 0.50 ms | 5.0 ms | **mongo-lite ×10** |
+| deleteOne + insert | 0.62 ms | 4.8 ms | **mongo-lite ×8** |
 | server RSS | 2.0 GB | 1.5 GB | mongodb (×0.7) |
 | kill -9 → reopen | 0.8 s | 1.3 s | **mongo-lite** |
 | db on disk | 1.0 GB | 96 MB | mongodb (compressed) |
@@ -206,15 +206,12 @@ The remaining losses are structural rather than incidental. Disk size is
 the big one: payloads are stored raw, so the log is 11x MongoDB's
 compressed files. RSS trails because every document carries its own arena.
 The range-scan gap is not the matcher — it is walking 65,536 documents
-that each live in a separate allocation, one pointer chase apiece. And
+that each live in a separate allocation, one pointer chase apiece.
-`sort` on `_id` still materializes candidates because nothing ordered
-covers `_id` yet; the same sort on an indexed field streams straight out
-of the index at 1.0 ms.
 
 Reproduce the table with `bash tests/e2e/compare-run.sh 1g 16k`; the
-pre-tree baseline is recorded in `tests/e2e/results/phase1.txt`, the run
+pre-tree baseline is recorded in `tests/e2e/results/phase1.txt`, and the
-above (with the B+tree, roadmap item 1) in
+runs with the B+tree and ordered `_id` index (roadmap items 1 and 2) in
-`tests/e2e/results/phase2.txt`.
+`tests/e2e/results/phase2.txt` and `tests/e2e/results/phase3.txt`.
 
 ### What is left (highest impact first)
 
@@ -226,15 +223,11 @@ traps in [ROADMAP.md](ROADMAP.md).
    highly compressible workloads massively; note Zig 0.16 ships zstd
    decompression only, and deflate would cap writes below the current
    insert rate.
-2. **An ordered `_id` index** — `sort({_id: ...})` still materializes every
+2. **Stop giving every document its own arena** — the source of both the
-   candidate, and integer `_id`s still scan. Both fall out of indexing the
-   encoded `_id`. The tree is in place, so an `_id` index update is a
-   leaf insert, not a tail memmove.
-3. **Stop giving every document its own arena** — the source of both the
    RSS gap and the range-scan gap. Storing canonical BSON bytes in a
    per-collection slab and matching against them (parsing only the fields a
    filter names) makes scans contiguous instead of a pointer chase.
-4. **Decompose the global lock** — one reader/writer lock covers the whole
+3. **Decompose the global lock** — one reader/writer lock covers the whole
    engine and is held across fsync, compaction and reply construction.
    Per-collection locks plus cross-connection group commit are the path to
    using more than one core on writes.
@@ -257,8 +250,16 @@ Done so far, with the measurement that drove each:
   records, no rebalancing on delete, and bulk bottom-up packing. Entry
   insertion and removal are a descent plus a leaf-local edit instead of a
   tail memmove, so writes into an already-built index stopped being
-  quadratic. `updateMany` 17.3 → 1.8 ms (2.8x slower than MongoDB → 3.7x
+  quadratic. `updateMany` 17.3 → 1.6 ms (2.8x slower than MongoDB → 4x
   faster); `createIndex` 62 → 51 ms.
+- **An ordered `_id` index** (roadmap item 2): every collection carries an
+  implicit `_id_` index (kept out of the secondary list, so the listing,
+  drop and log-format surfaces are unchanged; rebuilt after replay like
+  the secondaries). Its encoded keys are canonical, so the old
+  serialization-guarded docs-map fast path is gone and integer/string
+  `_id` point lookups, `$in` and ranges hit the tree instead of a full
+  scan. `sort({_id: ...})` is now an index-ordered scan with an early stop:
+  `sort+limit(20)` 6.2 → 2.4 ms (parity with MongoDB).
 - **Entry removal is a binary search**, not a scan of the whole index.
   `updateMany` 15.4 → 5.5 ms.
 - **Top-k sort selection** and an allocation-free decorate pass, plus

ROADMAP.md

@@ -1,9 +1,10 @@
 # Remaining performance work
 
-Status: **item 1 (B+tree over the encoded keys) is done** — landed and
+Status: **items 1 (B+tree over the encoded keys) and 2 (ordered `_id` index)
-verified in `tests/e2e/results/phase2.txt` (updateMany 17.3 → 1.8 ms,
+are done** — landed and verified in `tests/e2e/results/phase2.txt` and
-createIndex 62 → 51 ms). Its dependents (items 2 and 4) now stand on a
+`phase3.txt` (updateMany 17.3 → 1.6 ms, createIndex 62 → 51 ms, `_id`
-tree instead of a sorted array. Five items below, in dependency order.
+sort+limit 6.2 → 2.4 ms). Their dependents (items 4) now stand on a tree
+instead of a sorted array. Items below, in dependency order.
 Each is sized to be landed and verified on
 its own; the ordering constraints between them are the load-bearing part, so
 read those before picking one up.
@@ -101,7 +102,26 @@ differential. Both already exist and both are mutation-checked.
 
 ---
 
-## 2. Ordered `_id` index — **depends on 1**
+## 2. Ordered `_id` index — DONE
+
+Landed as an implicit `_id_` index on every `Collection` (a normal
+`index.Index` with keys `[_id: 1]`, kept out of the secondary `indexes`
+list so `listIndexes`/`dropIndexes`/`createIndex` and the log format are
+unchanged — no `index_create` record, no double listing). Maintained in
+`upsert` (through the same reserve-then-insert protocol as the
+secondaries) and `evict_doc`; rebuilt after replay by `build_all_indexes`
+alongside the secondaries. `index.plan` now takes it as a separate
+argument, so `{_id: ...}` equality, `$in` and ranges use the tree (the
+old serialization-guarded docs-map fast path — `plan_id`,
+`value_fast_path_safe` and friends — is deleted), and `sort({_id: ...})`
+becomes an index-ordered full scan with an early stop. A full `_id` scan
+cannot miss a document: every doc has an `_id` and the index is not
+sparse, and its keys are canonical (`bson.encode_key` gives int32 1,
+int64 1 and double 1.0 identical bytes).
+
+Recorded deltas vs `tests/e2e/results/phase2.txt`: `sort({_id:-1}).limit(20)`
+6.2 → 2.4 ms (2.3x slower than MongoDB → parity). Integer/string `_id`
+point lookups, `$in` and ranges no longer full-scan.
 
 **Why.** `sort({_id: ...})` still materializes every candidate, and integer
 `_id`s still fall back to a full collection scan on every `findOne`,
@@ -115,8 +135,8 @@ requires.
 
 Then delete `value_fast_path_safe` and friends (`src/index.zig`). They exist
 only because `serialize_value` gives `int32 1`, `int64 1` and `double 1.0`
-different bytes despite comparing equal. `bson.encode_key` already gives them
+different bytes despite comparing equal. `bson.encode_key` already gives
-identical bytes, so the guard is obsolete.
+them identical bytes, so the guard is obsolete.
 
 **Why it depends on item 1.** This index updates on *every* insert. Against a
 sorted array that is a tail memmove each time — roughly 51 GB of memmove over
@@ -128,7 +148,9 @@ only" caveat.
 
 **Watch.** A real `_id_` index may start appearing in `listIndexes` and
 writing an `index_create` record to the log. `e2e3.js` asserts on index
-listings — check it before assuming this is invisible.
+listings — check it before assuming this is invisible. (This landed without
+either: the index stays out of the secondary list, so the listing, drop and
+log surfaces are untouched; verified with `e2e3.js` unchanged.)
 
 ---
 

src/commands.zig

@@ -644,31 +644,10 @@ fn scan_sorted(
     const filter_doc = bson.Document{ .arena = undefined, .pairs = filter };
     var n: usize = 0;
 
-    // _id_ fast path: the docs map is the _id index. Skipped when the
+    // Index plan (the implicit _id_ index first, then the secondaries):
-    // queried value's compare-equivalence class is serialization-ambiguous
+    // candidates in index order, re-filtered. The returned ids alias the
-    // (see index.plan_id).
+    // docs map keys, valid under the read lock.
-    if (try index.plan_id(ctx.gpa, filter)) |id_plan| {
+    if (try index.plan(ctx.gpa, &coll.id_index, coll.indexes.items, filter, sort)) |p| {
-        var plan = id_plan;
-        defer plan.deinit(ctx.gpa);
-        // One scratch key, rebuilt per value: a key is never needed past its
-        // own lookup.
-        var key: std.ArrayListUnmanaged(u8) = .empty;
-        defer key.deinit(ctx.gpa);
-        for (plan.values) |v| {
-            key.clearRetainingCapacity();
-            try bson.write_serialized_value(v, ctx.gpa, &key);
-            const doc = coll.docs.get(key.items) orelse continue;
-            if (!try query.matches(ctx.gpa, &filter_doc, doc)) continue;
-            if (out) |list| try list.append(ctx.gpa, doc);
-            n += 1;
-            if (lim != 0 and n >= lim) break;
-        }
-        return n;
-    }
-
-    // Secondary-index plan: candidates in index order, re-filtered. The
-    // returned ids alias the docs map keys, valid under the read lock.
-    if (try index.plan(ctx.gpa, coll.indexes.items, filter, sort)) |p| {
         var plan = p;
         defer plan.deinit(ctx.gpa);
         var ids: std.ArrayListUnmanaged([]const u8) = .empty;

src/db.zig

@@ -12,10 +12,23 @@ const index = @import("index.zig");
 
 pub const Collection = struct {
     docs: std.StringHashMapUnmanaged(*bson.Document),
+    /// Secondary indexes (persisted through the log).
     indexes: std.ArrayListUnmanaged(index.Index),
+    /// The implicit _id_ index: every document has an _id and it is not
+    /// sparse, so entry count equals document count and a full scan of it
+    /// cannot miss a document — which is what the sort planner's full-scan
+    /// plan relies on. Kept out of `indexes` so the listing/drop commands
+    /// and the log format are unchanged (it is rebuilt on open like
+    /// everything else). `bson.encode_key` keys are canonical, so it also
+    /// replaces the old serialization-guarded docs-map fast path for
+    /// integer/string/etc. _id lookups.
+    id_index: index.Index,
 
-    fn init() Collection {
+    fn init(gpa: std.mem.Allocator) !Collection {
-        return .{ .docs = .empty, .indexes = .empty };
+        var self: Collection = .{ .docs = .empty, .indexes = .empty, .id_index = undefined };
+        const keys = [_]index.IndexKey{.{ .path = "_id", .descending = false }};
+        self.id_index = try index.Index.init(gpa, "_id_", &keys, false, false, null);
+        return self;
     }
 
     /// The secondary index with this name, or null. The single by-name
@@ -108,6 +121,7 @@ pub const Engine = struct {
         // Dropping a collection turns all of its records into garbage.
         self.live_docs -= coll.docs.count();
         self.dead_docs += coll.docs.count();
+        coll.id_index.deinit(self.gpa);
         for (coll.indexes.items) |*ix| ix.deinit(self.gpa);
         coll.indexes.deinit(self.gpa);
         var doc_it = coll.docs.iterator();
@@ -139,6 +153,7 @@ pub const Engine = struct {
     /// regenerating them from it, which is far cheaper than scanning.
     fn evict_doc(self: *Engine, coll: *Collection, id_key: []const u8) void {
         const old = coll.docs.fetchRemove(id_key) orelse return;
+        coll.id_index.remove_doc(self.gpa, old.value, old.key);
         for (coll.indexes.items) |*ix| ix.remove_doc(self.gpa, old.value, old.key);
         old.value.*.deinit();
         self.gpa.destroy(old.value);
@@ -241,6 +256,15 @@ pub const Engine = struct {
                 return err;
             };
         }
+        {
+            // The implicit _id_ index, through the same protocol: reserved
+            // before the log append, inserted infallibly after it.
+            var built = try coll.id_index.build_entries(self.gpa, owned, id_key);
+            built_list.append(self.gpa, .{ .built = built, .ix = &coll.id_index }) catch |err| {
+                built.deinit(self.gpa);
+                return err;
+            };
+        }
 
         // 2. The _id check, mirroring the pre-index behavior.
         if (mode == .insert and coll.docs.contains(id_key)) return error.DuplicateKey;
@@ -494,7 +518,9 @@ pub const Engine = struct {
         if (db.collections.getPtr(coll_name)) |coll| return coll;
         const coll_key = try self.gpa.dupe(u8, coll_name);
         errdefer self.gpa.free(coll_key);
-        try db.collections.put(self.gpa, coll_key, Collection.init());
+        var new_coll = try Collection.init(self.gpa);
+        errdefer new_coll.id_index.deinit(self.gpa);
+        try db.collections.put(self.gpa, coll_key, new_coll);
         return db.collections.getPtr(coll_name) orelse unreachable;
     }
 
@@ -624,8 +650,22 @@ pub const Engine = struct {
             var coll_it = db_entry.value_ptr.collections.iterator();
             while (coll_it.next()) |coll_entry| {
                 for (coll_entry.value_ptr.indexes.items) |*ix| {
-                    if (ix.count() > 0) continue; // defensive
+                    try self.rebuild_index(coll_entry.value_ptr, ix);
-                    var doc_it = coll_entry.value_ptr.docs.iterator();
+                }
+                try self.rebuild_index(coll_entry.value_ptr, &coll_entry.value_ptr.id_index);
+            }
+        }
+    }
+
+    /// Rebuild one index from the live documents. Runs after replay, so it
+    /// is order-independent; indexes already holding entries (maintained
+    /// live) are skipped defensively. A duplicate under a unique index logs
+    /// a loud warning and keeps the index (still correct as a candidate
+    /// generator; future writes are still enforced) — the database always
+    /// opens, leaving dropIndexes as an in-band recovery path.
+    fn rebuild_index(self: *Engine, coll: *Collection, ix: *index.Index) !void {
+        if (ix.count() > 0) return; // defensive
+        var doc_it = coll.docs.iterator();
         while (doc_it.next()) |doc_entry| {
             ix.append_doc_entries(self.gpa, doc_entry.value_ptr.*, doc_entry.key_ptr.*) catch |err| switch (err) {
                 error.ParallelArrays => {
@@ -640,9 +680,6 @@ pub const Engine = struct {
             std.debug.print("mongo-lite: WARNING: unique index '{s}' has duplicate keys in existing data; duplicates not enforced for existing documents\n", .{ix.name});
         }
     }
-            }
-        }
-    }
 
     /// Register an (empty) index from a persisted spec document. A repeated
     /// create record for the same name is an idempotent no-op.
@@ -721,6 +758,8 @@ fn apply_record(ctx: *anyopaque, record: storage.Record, doc: *bson.Document) an
             self.live_docs += 1;
             key_owned = true;
             stored = true;
+            // The _id_ entry is added after replay, in build_all_indexes,
+            // together with the secondary indexes.
         },
         storage.record_type_delete => self.evict_doc(coll, id_key),
         else => {},

src/index.zig

@@ -1523,13 +1523,32 @@ pub const Plan = struct {
 /// documents: the one covering the longest leading run of equality/$in
 /// predicates, optionally with a range on the next key. Returns null when
 /// nothing usable remains — the caller scans.
-pub fn plan(gpa: std.mem.Allocator, indexes: []const Index, filter: []const bson.Pair, sort: []const query.SortKey) !?Plan {
+///
-    if (indexes.len == 0) return null;
+/// `id_ix` is the collection's implicit `_id_` index (kept separate from
+/// the secondary `indexes` list so the listing/drop commands stay
+/// unchanged). Every document has an `_id` and the index is not sparse, so
+/// a full scan of it cannot miss a document — which is what the sort
+/// planner's full-scan plan relies on. The encoded keys are canonical, so
+/// `_id` equality on compare-equal values (int32 1, int64 1, double 1.0)
+/// finds the same entries.
+pub fn plan(
+    gpa: std.mem.Allocator,
+    id_ix: ?*const Index,
+    indexes: []const Index,
+    filter: []const bson.Pair,
+    sort: []const query.SortKey,
+) !?Plan {
+    if (id_ix == null and indexes.len == 0) return null;
     var clauses: std.ArrayListUnmanaged(Clause) = .empty;
     defer clauses.deinit(gpa);
     try flatten_clauses(gpa, filter, &clauses);
 
     var best: ?Plan = null;
+    if (id_ix) |idx| {
+        if (try evaluate_index(gpa, idx, clauses.items, sort)) |cand| {
+            best = cand;
+        }
+    }
     for (indexes) |*ix| {
         var cand = (try evaluate_index(gpa, ix, clauses.items, sort)) orelse continue;
         if (best) |b| {
@@ -1682,112 +1701,6 @@ fn evaluate_index(gpa: std.mem.Allocator, ix: *const Index, clauses: []const Cla
     return pl;
 }
 
-// ---------------------------------------------------------------------------
-// _id fast path
-// ---------------------------------------------------------------------------
-
-pub const IdPlan = struct {
-    /// Values to look up in the docs map. A single $eq value is a heap copy
-    /// owned by this plan; a $in list aliases the filter's array. Freed by
-    /// deinit.
-    values: []const bson.Value,
-    owned: bool,
-
-    pub fn deinit(self: *IdPlan, gpa: std.mem.Allocator) void {
-        if (self.owned) gpa.free(self.values);
-    }
-};
-
-/// Plan for the implicit _id_ index (the docs map). Only applies to a
-/// top-level _id equality/$eq/$in clause, and only when the queried value's
-/// compare-equivalence class is serialization-canonical. bson.compare calls
-/// int32 1, int64 1 and double 1.0 equal (and string/symbol/code "x" equal,
-/// and nested variants), but serialize_value produces different map keys —
-/// a hash lookup would then miss documents a scan would match.
-///
-/// The returned plan must be deinit'd: the single-value case is heap-copied
-/// so the slice never points into a temporary (ReleaseFast reuses the stack,
-/// which turned a dangling anonymous-list pointer into garbage).
-pub fn plan_id(gpa: std.mem.Allocator, filter: []const bson.Pair) error{OutOfMemory}!?IdPlan {
-    // The first usable _id clause wins; no flattening buffer is needed
-    // because nothing is compared across clauses. $and members are searched
-    // like top-level pairs, every other operator skipped — same rule as
-    // flatten_clauses, and safe for the same reason (the full filter is
-    // re-applied to every candidate).
-    for (filter) |p| {
-        if (p.key.len > 0 and p.key[0] == '$') {
-            if (!std.mem.eql(u8, p.key, "$and")) continue;
-            const members = switch (p.value) {
-                .array => |a| a,
-                else => continue,
-            };
-            for (members) |m| {
-                const mp = switch (m) {
-                    .doc => |d| d,
-                    else => continue,
-                };
-                if (try plan_id(gpa, mp)) |found| return found;
-            }
-            continue;
-        }
-        if (!std.mem.eql(u8, p.key, "_id")) continue;
-        if (try id_lookup_values(gpa, p.value)) |lookup| return .{ .values = lookup.values, .owned = lookup.owned };
-    }
-    return null;
-}
-
-const IdLookup = struct {
-    values: []const bson.Value,
-    owned: bool,
-};
-
-/// The map-lookup values for one _id clause, or null when it is not a pure
-/// equality/$in of fast-path-safe values. A range is unusable here: the docs
-/// map is a hash, not an ordered structure.
-fn id_lookup_values(gpa: std.mem.Allocator, v: bson.Value) error{OutOfMemory}!?IdLookup {
-    var info = CompInfo{};
-    analyze_clause(v, &info);
-    if (info.lo != null or info.hi != null) return null;
-    if (info.eq) |e| {
-        if (!value_fast_path_safe(e)) return null;
-        // Heap-copy the single value: a pointer to a stack or anonymous
-        // temporary would dangle once this frame returns.
-        const buf = try gpa.alloc(bson.Value, 1);
-        buf[0] = e;
-        return .{ .values = buf[0..1], .owned = true };
-    }
-    if (info.in_values) |list| {
-        for (list) |m| {
-            if (!value_fast_path_safe(m)) return null;
-        }
-        return .{ .values = list, .owned = false };
-    }
-    return null;
-}
-
-/// Whether serialize_value is injective on the value's compare-equivalence
-/// class. False for numbers (int32 1 / int64 1 / double 1.0), string/symbol/
-/// code (compared equal, serialized with different type tags), opaque
-/// values (compared by payload only), and anything containing those.
-fn value_fast_path_safe(v: bson.Value) bool {
-    return switch (v) {
-        .double, .int32, .int64, .string, .symbol, .code, .opaque_val => false,
-        .doc => |pairs| blk: {
-            for (pairs) |p| {
-                if (!value_fast_path_safe(p.value)) break :blk false;
-            }
-            break :blk true;
-        },
-        .array => |items| blk: {
-            for (items) |it| {
-                if (!value_fast_path_safe(it)) break :blk false;
-            }
-            break :blk true;
-        },
-        else => true,
-    };
-}
-
 // ---------------------------------------------------------------------------
 // Tests
 // ---------------------------------------------------------------------------
@@ -2301,63 +2214,6 @@ test "lookup_range matches a brute-force filter over random data" {
     }
 }
 
-test "id fast path guards and $in" {
-    // plan_id may heap-copy the single value; run it through the allocator
-    // and free. The helper asserts on whether a plan was produced.
-    const gpa = testing.allocator;
-    const plans = struct {
-        fn has(pairs: []const bson.Pair) !bool {
-            var p = try plan_id(gpa, pairs);
-            defer if (p) |*pl| pl.deinit(gpa);
-            return p != null;
-        }
-    };
-    // Numbers never use the fast path (compare-equal but serialize-different).
-    try testing.expect(!try plans.has(&.{.{ .key = "_id", .value = .{ .int32 = 1 } }}));
-    // Strings are skipped too: a stored symbol/code _id compare-equals a
-    // string query but serializes differently.
-    try testing.expect(!try plans.has(&.{.{ .key = "_id", .value = .{ .string = "x" } }}));
-    // Truly canonical values (bool, ObjectId) do use it.
-    try testing.expect(try plans.has(&.{.{ .key = "_id", .value = .{ .bool = true } }}));
-    const oid = [_]u8{ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 };
-    try testing.expect(try plans.has(&.{.{ .key = "_id", .value = .{ .object_id = oid } }}));
-    // $in with a number member is skipped.
-    const mixed = [_]bson.Pair{.{ .key = "_id", .value = .{ .doc = &.{
-        .{ .key = "$in", .value = .{ .array = &.{ .{ .bool = true }, .{ .int32 = 1 } } } },
-    } } }};
-    try testing.expect(!try plans.has(&mixed));
-    const safe_in = [_]bson.Pair{.{ .key = "_id", .value = .{ .doc = &.{
-        .{ .key = "$in", .value = .{ .array = &.{ .{ .bool = true }, .{ .bool = false } } } },
-    } } }};
-    try testing.expect(try plans.has(&safe_in));
-    // $and members count as top-level.
-    const and_f = [_]bson.Pair{.{ .key = "$and", .value = .{ .array = &.{
-        .{ .doc = &.{.{ .key = "_id", .value = .{ .bool = true } }} },
-    } } }};
-    try testing.expect(try plans.has(&and_f));
-    // $or is not usable for the fast path.
-    const or_f = [_]bson.Pair{.{ .key = "$or", .value = .{ .array = &.{
-        .{ .doc = &.{.{ .key = "_id", .value = .{ .bool = true } }} },
-    } } }};
-    try testing.expect(!try plans.has(&or_f));
-    // A doc containing a number is not fast-path safe.
-    const doc_id = [_]bson.Pair{.{ .key = "_id", .value = .{ .doc = &.{.{ .key = "n", .value = .{ .int32 = 1 } }} } }};
-    try testing.expect(!try plans.has(&doc_id));
-    // A doc containing a string is unsafe too (string/symbol/code class).
-    const doc_str = [_]bson.Pair{.{ .key = "_id", .value = .{ .doc = &.{.{ .key = "n", .value = .{ .string = "x" } }} } }};
-    try testing.expect(!try plans.has(&doc_str));
-    // A doc of canonical values is safe.
-    const doc_safe = [_]bson.Pair{.{ .key = "_id", .value = .{ .doc = &.{.{ .key = "n", .value = .{ .bool = true } }} } }};
-    try testing.expect(try plans.has(&doc_safe));
-    // The single-value plan owns a usable copy: the ObjectId must read back
-    // exactly (this is the ReleaseFast regression the fix guards — a dangling
-    // pointer read back as garbage).
-    var p = try plan_id(gpa, &.{.{ .key = "_id", .value = .{ .object_id = oid } }});
-    defer if (p) |*pl| pl.deinit(gpa);
-    try testing.expect(p != null);
-    try testing.expectEqualSlices(u8, &oid, &p.?.values[0].object_id);
-}
-
 test "TTL spec round-trips through write_spec and compares in spec_equal" {
     const gpa = testing.allocator;
     const spec = doc_of(&.{
@@ -2471,6 +2327,93 @@ test "TTL spec rejects compound keys and bad expireAfterSeconds" {
     try testing.expectEqualStrings("expireAt_1", ix.name);
 }
 
+test "the _id index plan covers equality, ranges and _id sort order" {
+    // The implicit _id_ index is a normal Index (keys = [_id: 1]) passed to
+    // plan separately from the secondaries. Its encoded keys are canonical,
+    // so compare-equal numbers (int32/int64/double) find the same entries,
+    // and a full scan of it is the sort planner's order supply for
+    // sort({_id: ...}).
+    const gpa = testing.allocator;
+    var ix = try simple_index(gpa, &.{"_id"}, false, false);
+    defer ix.deinit(gpa);
+    for (0..5) |i| {
+        const d = doc_of(&.{
+            .{ .key = "_id", .value = .{ .int32 = @intCast(i + 1) } },
+            .{ .key = "v", .value = .{ .int32 = @intCast(i) } },
+        });
+        const id = try std.fmt.allocPrint(gpa, "d{d}", .{i + 1});
+        defer gpa.free(id);
+        _ = try ix.add_doc(gpa, &d, id, false);
+    }
+
+    // {_id: 3} → an equality plan whose candidates are just that doc.
+    {
+        const f = [_]bson.Pair{.{ .key = "_id", .value = .{ .int32 = 3 } }};
+        var p = (try plan(gpa, &ix, &.{}, &f, &.{})).?;
+        defer p.deinit(gpa);
+        try testing.expectEqual(@as(usize, 1), p.key_len());
+        var ids: std.ArrayListUnmanaged([]const u8) = .empty;
+        defer ids.deinit(gpa);
+        try p.search(gpa, &ids);
+        try testing.expectEqual(@as(usize, 1), ids.items.len);
+        try testing.expectEqualStrings("d3", ids.items[0]);
+    }
+    // The same query as an int64 and a double finds the int32 entry.
+    {
+        const f = [_]bson.Pair{.{ .key = "_id", .value = .{ .int64 = 3 } }};
+        var p = (try plan(gpa, &ix, &.{}, &f, &.{})).?;
+        defer p.deinit(gpa);
+        var ids: std.ArrayListUnmanaged([]const u8) = .empty;
+        defer ids.deinit(gpa);
+        try p.search(gpa, &ids);
+        try testing.expectEqual(@as(usize, 1), ids.items.len);
+        try testing.expectEqualStrings("d3", ids.items[0]);
+    }
+    // A range on _id yields the band in index order.
+    {
+        const f = [_]bson.Pair{.{ .key = "_id", .value = .{ .doc = &.{.{ .key = "$gte", .value = .{ .int32 = 3 } }} } }};
+        var p = (try plan(gpa, &ix, &.{}, &f, &.{})).?;
+        defer p.deinit(gpa);
+        var ids: std.ArrayListUnmanaged([]const u8) = .empty;
+        defer ids.deinit(gpa);
+        try p.search(gpa, &ids);
+        try testing.expectEqual(@as(usize, 3), ids.items.len);
+        try testing.expectEqualStrings("d3", ids.items[0]);
+        try testing.expectEqualStrings("d5", ids.items[2]);
+    }
+    // sort({_id: 1}) with no filter: a full index scan that supplies the
+    // order — the plan the sort planner needs to stop materializing.
+    {
+        const sort = [_]query.SortKey{.{ .path = "_id", .descending = false }};
+        var p = (try plan(gpa, &ix, &.{}, &.{}, &sort)).?;
+        defer p.deinit(gpa);
+        try testing.expect(p.provides_sort and !p.backward);
+        var ids: std.ArrayListUnmanaged([]const u8) = .empty;
+        defer ids.deinit(gpa);
+        try p.search(gpa, &ids);
+        try testing.expectEqual(@as(usize, 5), ids.items.len);
+        try testing.expectEqualStrings("d1", ids.items[0]);
+        try testing.expectEqualStrings("d5", ids.items[4]);
+    }
+    // sort({_id: -1}): backward.
+    {
+        const sort = [_]query.SortKey{.{ .path = "_id", .descending = true }};
+        var p = (try plan(gpa, &ix, &.{}, &.{}, &sort)).?;
+        defer p.deinit(gpa);
+        try testing.expect(p.provides_sort and p.backward);
+        var ids: std.ArrayListUnmanaged([]const u8) = .empty;
+        defer ids.deinit(gpa);
+        try p.search(gpa, &ids);
+        try testing.expectEqualStrings("d5", ids.items[0]);
+        try testing.expectEqualStrings("d1", ids.items[4]);
+    }
+    // A filter naming no _id field leaves the _id index unusable.
+    {
+        const f = [_]bson.Pair{.{ .key = "v", .value = .{ .int32 = 1 } }};
+        try testing.expect((try plan(gpa, &ix, &.{}, &f, &.{})) == null);
+    }
+}
+
 test "planner picks eq run, ranges, and bails on sparse null" {
     const gpa = testing.allocator;
     var ix = try simple_index(gpa, &.{ "a", "b" }, false, false);
@@ -2484,7 +2427,7 @@ test "planner picks eq run, ranges, and bails on sparse null" {
             .{ .key = "a", .value = .{ .int32 = 1 } },
             .{ .key = "b", .value = .{ .int32 = 2 } },
         };
-        var p = (try plan(gpa, &.{ix}, &f, &.{})).?;
+        var p = (try plan(gpa, null, &.{ix}, &f, &.{})).?;
         defer p.deinit(gpa);
         try testing.expectEqual(@as(usize, 2), p.key_len());
         try testing.expect(p.lo == null and p.hi == null);
@@ -2495,7 +2438,7 @@ test "planner picks eq run, ranges, and bails on sparse null" {
             .{ .key = "a", .value = .{ .int32 = 1 } },
             .{ .key = "b", .value = .{ .doc = &.{.{ .key = "$gt", .value = .{ .int32 = 2 } }} } },
         };
-        var p = (try plan(gpa, &.{ix}, &f, &.{})).?;
+        var p = (try plan(gpa, null, &.{ix}, &f, &.{})).?;
         defer p.deinit(gpa);
         try testing.expectEqual(@as(usize, 1), p.key_len());
         try testing.expect(p.hi == null and p.lo != null and !p.lo_incl);
@@ -2503,14 +2446,14 @@ test "planner picks eq run, ranges, and bails on sparse null" {
     // {a: 1} only → prefix run of 1.
     {
         const f = [_]bson.Pair{.{ .key = "a", .value = .{ .int32 = 1 } }};
-        var p = (try plan(gpa, &.{ix}, &f, &.{})).?;
+        var p = (try plan(gpa, null, &.{ix}, &f, &.{})).?;
         defer p.deinit(gpa);
         try testing.expectEqual(@as(usize, 1), p.key_len());
     }
     // Pure range on the first key → key_len 0 with a bound.
     {
         const f = [_]bson.Pair{.{ .key = "a", .value = .{ .doc = &.{.{ .key = "$gte", .value = .{ .int32 = 1 } }} } }};
-        var p = (try plan(gpa, &.{ix}, &f, &.{})).?;
+        var p = (try plan(gpa, null, &.{ix}, &f, &.{})).?;
         defer p.deinit(gpa);
         try testing.expectEqual(@as(usize, 0), p.key_len());
         try testing.expect(p.lo != null and p.lo_incl);
@@ -2518,23 +2461,23 @@ test "planner picks eq run, ranges, and bails on sparse null" {
     // Unusable filter → no plan.
     {
         const f = [_]bson.Pair{.{ .key = "a", .value = .{ .doc = &.{.{ .key = "$regex", .value = .{ .string = "^x" } }} } }};
-        try testing.expect((try plan(gpa, &.{ix}, &f, &.{})) == null);
+        try testing.expect((try plan(gpa, null, &.{ix}, &f, &.{})) == null);
         const or_f = [_]bson.Pair{.{ .key = "$or", .value = .{ .array = &.{
             .{ .doc = &.{.{ .key = "a", .value = .{ .int32 = 1 } }} },
         } } }};
-        try testing.expect((try plan(gpa, &.{ix}, &or_f, &.{})) == null);
+        try testing.expect((try plan(gpa, null, &.{ix}, &or_f, &.{})) == null);
     }
     // Sparse index bails on a null component.
     {
         const f = [_]bson.Pair{.{ .key = "a", .value = .null }};
-        try testing.expect((try plan(gpa, &.{sp}, &f, &.{})) == null);
+        try testing.expect((try plan(gpa, null, &.{sp}, &f, &.{})) == null);
         // Non-sparse is fine with null.
-        var p = (try plan(gpa, &.{ix}, &f, &.{})).?;
+        var p = (try plan(gpa, null, &.{ix}, &f, &.{})).?;
         defer p.deinit(gpa);
         try testing.expect(p.key_len() == 1);
         // A null inside $in bails too.
         const fin = [_]bson.Pair{.{ .key = "a", .value = .{ .doc = &.{.{ .key = "$in", .value = .{ .array = &.{ .{ .int32 = 1 }, .null } } }} } }};
-        try testing.expect((try plan(gpa, &.{sp}, &fin, &.{})) == null);
+        try testing.expect((try plan(gpa, null, &.{sp}, &fin, &.{})) == null);
     }
     // $in cartesian product is capped.
     {
@@ -2545,6 +2488,6 @@ test "planner picks eq run, ranges, and bails on sparse null" {
             .{ .key = "b", .value = .{ .doc = &.{.{ .key = "$in", .value = .{ .array = &members } }} } },
         };
         // 20 * 20 = 400 > 100 → fall back to a scan.
-        try testing.expect((try plan(gpa, &.{ix}, &f, &.{})) == null);
+        try testing.expect((try plan(gpa, null, &.{ix}, &f, &.{})) == null);
     }
 }

tests/e2e/results/phase3.txt

@@ -0,0 +1,38 @@
+# Phase 3 gate — mongo-lite vs MongoDB 8.3.7, 1g dataset / ~16k docs
+# Ratio < 1.0 = mongo-lite faster. Reproduce: bash tests/e2e/compare-run.sh 1g 16k
+# This run includes roadmap items 1 (B+tree) and 2 (ordered _id index).
+# Compare: tests/e2e/results/phase1.txt (pre-tree baseline).
+
+benchmark                                   mongo-lite              mongodb                 ratio
+insertOne (sequential) ×200                 0.19 ms                 4.4 ms                  0.0x
+bulk insert throughput                      816.5 MB/s              716.8 MB/s              1.1x
+docs loaded                                 65,536                  65,536                  1.0x
+createIndex({k: 1})                         50.7 ms                 76.8 ms                 0.7x
+countDocuments({})                          1.5 ms                  11.1 ms                 0.1x
+findOne({_id: <ObjectId>})                  0.57 ms                 0.64 ms                 0.9x
+findOne({k: 500}) (indexed)                 0.58 ms                 1.5 ms                  0.4x
+find({p: {$gte,$lt}}).count() (scan)        22.1 ms                 12.9 ms                 1.7x
+find({}).sort({_id:-1}).limit(20)           2.4 ms                  2.2 ms                  1.1x
+find({}, {proj}).limit(1000)                3.7 ms                  4.3 ms                  0.9x
+aggregate $group by k                       9.1 ms                  12.5 ms                 0.7x
+updateOne({_id}) ×50                        0.16 ms                 0.19 ms                 0.8x
+updateMany({k: 7}, {$inc})                  1.6 ms                  6.4 ms                  0.3x
+deleteOne({_id}) + insertOne                0.62 ms                 4.8 ms                  0.1x
+node client RSS                             159 MB                  156 MB                  1.0x
+server RSS                                  1973 MB                 1334 MB
+kill -9 reopen                              0.8s                    1.3s
+db on disk                                  1025MB                  91MB
+
+# Item 2 (ordered _id index) deltas vs phase2:
+#   sort({_id:-1}).limit(20)  6.2 -> 2.4 ms  (2.3x slower than mongod -> parity):
+#                 the sort planner now scans the _id tree in order and stops
+#                 at the page limit instead of materializing every candidate.
+#   integer/string _id findOne, updateOne, deleteOne no longer fall back to a
+#                 full collection scan (verified separately: point lookups,
+#                 $in, ranges, int64/int32 compare-equal equality all hit the
+#                 tree; the docs-map serialization-guard fast path is gone).
+#
+# Remaining gaps and where they are addressed:
+#   db on disk   11x  -> Phase 3 (block-compressed log)
+#   range-scan  1.7x  -> Phase 4 (contiguous byte storage, not the matcher)
+#   server RSS  1.5x  -> Phase 4 (per-document arena -> byte storage)