dev/MultiforaDB
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

index: B+tree over the encoded keys (roadmap item 1)

Browse Source 
Replace Index.entries (one sorted array) with a B+tree so writes into an
already-built index stop being quadratic. Nodes are fixed 4 KiB slotted
pages in a flat u32-addressed ArrayListUnmanaged(Node); records longer
than a quarter page spill to an append-only overflow slab (BSON strings
reach 16 MB). Leaves are doubly linked for ordered iteration; the flat
node array stays one contiguous byte range for a later checkpoint.

Insertion descends by separator and splits leaves/internals upward,
promoting keys via a stable copy (a nested split can otherwise clobber
the promoted-key scratch). Deletion does not rebalance: emptied leaves
are unlinked and dropped from their parent, internal nodes may carry one
child, and dead pages are abandoned in place (node memory peaks at the
tree's peak size, exactly what the old array's capacity did). Lookups
are lower-bound seeks plus leaf-chain band scans, so equal keys may
span leaves freely. Bulk build (append_doc_entries + finish_bulk) sorts
a staging array and packs leaves bottom-up. reserve_for now takes the
built entries and reserves exact overflow bytes plus a worst-case node
count, keeping insert_entries infallible after the log append.

db.zig: TTL sweep now seeks the minimum-datetime encoded key and walks
the contiguous datetime band, stopping at the cutoff or type change.

Measured (tests/e2e/results/phase2.txt): updateMany 17.3 -> 1.8 ms
(2.8x slower than MongoDB -> 3.7x faster), createIndex 62 -> 51 ms.

Verified: unit suite ReleaseFast/ReleaseSafe/Debug (incl. the existing
lookup_range and remove_doc differentials, plus a new incremental
insert/remove differential against a brute-force model), the crash pair,
e2e3/e2e4/e2e6, and dev stress tests for depth-2 splits, full drains,
and spilled records through internal levels.
This commit is contained in:
Alexey Shakhmatov
2026-08-02 21:10:25 +03:00
parent 71112b0ff7
commit 61fe952125
8 changed files with 1450 additions and 218 deletions
Download Patch File Download Diff File Expand all files Collapse all files

56
README.md
View File

@@ -185,22 +185,22 @@ With the ReleaseFast default build (MongoDB 8.3.7 on the same Mac):
| benchmark | mongo-lite | mongodb | winner |
|---|---|---|---|
| insertOne (sequential) | 0.19 ms | 5.0 ms | **mongo-lite ×26** |
| bulk insert (insertMany) | 853 MB/s | 690 MB/s | **mongo-lite ×1.2** |
| createIndex({k: 1}) | 62 ms | 78 ms | **mongo-lite** |
| countDocuments({}) | 1.5 ms | 11.5 ms | **mongo-lite ×8** |
| findOne({_id}) | 0.48 ms | 0.54 ms | mongo-lite |
| findOne indexed | 0.64 ms | 4.3 ms | **mongo-lite ×7** |
| range-scan count | 21 ms | 13 ms | mongodb ×1.7 |
| sort + limit(20), on `_id` | 4.3 ms | 2.0 ms | mongodb ×2 |
| insertOne (sequential) | 0.19 ms | 4.1 ms | **mongo-lite ×22** |
| bulk insert (insertMany) | 810 MB/s | 714 MB/s | **mongo-lite ×1.1** |
| createIndex({k: 1}) | 51 ms | 82 ms | **mongo-lite** |
| countDocuments({}) | 1.5 ms | 13.8 ms | **mongo-lite ×9** |
| findOne({_id}) | 0.57 ms | 0.67 ms | mongo-lite |
| findOne indexed | 0.57 ms | 1.8 ms | **mongo-lite ×3** |
| range-scan count | 20 ms | 13 ms | mongodb ×1.6 |
| sort + limit(20), on `_id` | 6.2 ms | 2.7 ms | mongodb ×2.3 |
| sort + limit(20), indexed field | 1.0 ms | — | — |
| aggregate $group | 9.8 ms | 13.7 ms | **mongo-lite** |
| updateOne({_id}) | 0.16 ms | 0.19 ms | mongo-lite |
| updateMany (65 docs) | 5.5 ms | 6.1 ms | mongo-lite |
| deleteOne + insert | 0.62 ms | 4.9 ms | **mongo-lite ×8** |
| server RSS | 2.0 GB | 1.4 GB | mongodb (×0.7) |
| aggregate $group | 11.5 ms | 15.5 ms | **mongo-lite** |
| updateOne({_id}) | 0.17 ms | 0.19 ms | mongo-lite |
| updateMany (65 docs) | 1.8 ms | 6.7 ms | **mongo-lite ×3.7** |
| deleteOne + insert | 0.50 ms | 5.0 ms | **mongo-lite ×10** |
| server RSS | 2.0 GB | 1.5 GB | mongodb (×0.7) |
| kill -9 → reopen | 0.8 s | 1.3 s | **mongo-lite** |
| db on disk | 1.0 GB | 93 MB | mongodb (compressed) |
| db on disk | 1.0 GB | 96 MB | mongodb (compressed) |
The remaining losses are structural rather than incidental. Disk size is
the big one: payloads are stored raw, so the log is 11x MongoDB's
@@ -211,8 +211,10 @@ that each live in a separate allocation, one pointer chase apiece. And
covers `_id` yet; the same sort on an indexed field streams straight out
of the index at 1.0 ms.
Reproduce the table with `bash tests/e2e/compare-run.sh 1g 16k`; the run
above is recorded in `tests/e2e/results/phase1.txt`.
Reproduce the table with `bash tests/e2e/compare-run.sh 1g 16k`; the
pre-tree baseline is recorded in `tests/e2e/results/phase1.txt`, the run
above (with the B+tree, roadmap item 1) in
`tests/e2e/results/phase2.txt`.
### What is left (highest impact first)
@@ -224,20 +226,15 @@ traps in [ROADMAP.md](ROADMAP.md).
highly compressible workloads massively; note Zig 0.16 ships zstd
decompression only, and deflate would cap writes below the current
insert rate.
2. **A B-tree over the encoded keys** — entry insert still memmoves the
tail of a sorted array, so writing into a collection that already has an
index is quadratic. A flat, `u32`-indexed node array would also be
dumpable into a checkpoint, which is what makes a fast reopen possible.
3. **An ordered `_id` index** — `sort({_id: ...})` still materializes every
2. **An ordered `_id` index** — `sort({_id: ...})` still materializes every
candidate, and integer `_id`s still scan. Both fall out of indexing the
encoded `_id`. It wants the tree first: an `_id` index updates on every
insert, and doing that against a sorted array is only cheap because
ObjectIds append at the end.
4. **Stop giving every document its own arena** — the source of both the
encoded `_id`. The tree is in place, so an `_id` index update is a
leaf insert, not a tail memmove.
3. **Stop giving every document its own arena** — the source of both the
RSS gap and the range-scan gap. Storing canonical BSON bytes in a
per-collection slab and matching against them (parsing only the fields a
filter names) makes scans contiguous instead of a pointer chase.
5. **Decompose the global lock** — one reader/writer lock covers the whole
4. **Decompose the global lock** — one reader/writer lock covers the whole
engine and is held across fsync, compaction and reply construction.
Per-collection locks plus cross-connection group commit are the path to
using more than one core on writes.
@@ -255,6 +252,13 @@ Done so far, with the measurement that drove each:
array. `createIndex` over 65,536 documents 649 → 44 ms.
- **Index entries hold encoded byte keys**, so comparing them is a memcmp
rather than a walk over values in unrelated arenas.
- **A B+tree over the encoded keys** (roadmap item 1): fixed 4 KiB slotted
pages in a flat u32-addressed node array, an overflow slab for long
records, no rebalancing on delete, and bulk bottom-up packing. Entry
insertion and removal are a descent plus a leaf-local edit instead of a
tail memmove, so writes into an already-built index stopped being
quadratic. `updateMany` 17.3 → 1.8 ms (2.8x slower than MongoDB → 3.7x
faster); `createIndex` 62 → 51 ms.
- **Entry removal is a binary search**, not a scan of the whole index.
`updateMany` 15.4 → 5.5 ms.
- **Top-k sort selection** and an allocation-free decorate pass, plus

29
ROADMAP.md
View File

@@ -1,6 +1,10 @@
# Remaining performance work
Five items, in dependency order. Each is sized to be landed and verified on
Status: **item 1 (B+tree over the encoded keys) is done** — landed and
verified in `tests/e2e/results/phase2.txt` (updateMany 17.3 → 1.8 ms,
createIndex 62 → 51 ms). Its dependents (items 2 and 4) now stand on a
tree instead of a sorted array. Five items below, in dependency order.
Each is sized to be landed and verified on
its own; the ordering constraints between them are the load-bearing part, so
read those before picking one up.
@@ -43,7 +47,28 @@ touching indexes needs `e2e3.js` and `e2e4.js`.
---
## 1. B-tree over the encoded keys
---
## 1. B-tree over the encoded keys — DONE
Landed as a B+tree in `src/index.zig`: fixed 4 KiB slotted pages in a flat
u32-addressed `ArrayListUnmanaged(Node)`, an append-only overflow slab for
records longer than a quarter page (BSON strings reach 16 MB), leaves
linked for ordered iteration, bulk bottom-up packing for
`append_doc_entries` + `finish_bulk`, `remove_doc` regenerating entries
and removing them with a descent plus a leaf-local edit, and no
rebalancing on delete (empty leaves are unlinked and dropped; internal
nodes may carry one child). The flat node array stays the contiguous byte
range that item 4 can write into a checkpoint. Deletes abandon dead pages
rather than reusing them, so node memory peaks at the tree's peak size,
exactly what the old entry array's capacity did.
Recorded deltas vs `tests/e2e/results/phase1.txt`: `updateMany` 17.3 →
1.8 ms (2.8x slower than MongoDB → 3.7x faster), `createIndex` 62.4 →
50.8 ms. Verified with `zig build test` (ReleaseFast and ReleaseSafe), the
crash pair, `e2e3.js`/`e2e4.js`/`e2e6.js`, plus a 50k-entry stress (bulk
build, random inserts, random deletes, full drain) and a spill stress
(2 KiB keys through splits and internal nodes).
**Why.** Index entries live in one sorted array, so inserting an entry
memmoves the tail. Building an index is fine (entries are appended and sorted

36
src/db.zig
View File

@@ -255,10 +255,10 @@ pub const Engine = struct {
};
}
// 4. Reserve entry capacity — the last fallible step, so the entry
// 4. Reserve tree capacity — the last fallible step, so the entry
// insertion after the log append is infallible.
for (built_list.items) |*b| {
try b.ix.reserve_for(self.gpa, b.built.entries.items.len);
try b.ix.reserve_for(self.gpa, b.built.entries.items);
}
// 5. Log (and sync) before anything becomes visible.
@@ -365,7 +365,7 @@ pub const Engine = struct {
while (doc_it.next()) |entry| {
try ix.append_doc_entries(self.gpa, entry.value_ptr.*, entry.key_ptr.*);
}
_ = try ix.finish_bulk(true);
_ = try ix.finish_bulk(self.gpa, true);
// Reserve the collection slot, then persist and publish.
try coll.indexes.ensureUnusedCapacity(self.gpa, 1);
@@ -428,16 +428,16 @@ pub const Engine = struct {
for (coll_entry.value_ptr.indexes.items) |*ix| {
const ttl = ix.ttl orelse continue;
const cutoff: i128 = @as(i128, now_ms) - @as(i128, ttl) * 1000;
for (ix.entries.items) |e| {
// Still a linear walk: the type test cannot be a
// one-sided range lookup, because bson compare order
// ranks datetime above null, numbers and strings, so
// a datetime upper bound would also select every
// value of a lesser type. (Datetimes are contiguous
// in that order, so a two-sided band lookup would
// work — that comes with the tree.)
const ms = bson.encoded_leading_datetime(e.key) orelse continue;
if (@as(i128, ms) > cutoff) continue;
// bson compare order ranks datetime above null, numbers
// and strings and below only timestamp and maxKey, so
// datetimes form a contiguous band in the encoded key
// order: seek the minimum datetime and stop when the
// leading type changes or the cutoff is passed.
const min_dt = [_]u8{ bson.encoded_datetime_tag, 0, 0, 0, 0, 0, 0, 0, 0 };
var it = ix.seek(&min_dt);
while (it.next()) |e| {
const ms = bson.encoded_leading_datetime(e.key) orelse break;
if (@as(i128, ms) > cutoff) break;
try ids.append(self.gpa, try self.gpa.dupe(u8, e.id));
}
}
@@ -624,7 +624,7 @@ pub const Engine = struct {
var coll_it = db_entry.value_ptr.collections.iterator();
while (coll_it.next()) |coll_entry| {
for (coll_entry.value_ptr.indexes.items) |*ix| {
if (ix.entries.items.len > 0) continue; // defensive
if (ix.count() > 0) continue; // defensive
var doc_it = coll_entry.value_ptr.docs.iterator();
while (doc_it.next()) |doc_entry| {
ix.append_doc_entries(self.gpa, doc_entry.value_ptr.*, doc_entry.key_ptr.*) catch |err| switch (err) {
@@ -636,7 +636,7 @@ pub const Engine = struct {
};
}
// Tolerated, not enforced: the database must always open.
if (try ix.finish_bulk(false)) {
if (try ix.finish_bulk(self.gpa, false)) {
std.debug.print("mongo-lite: WARNING: unique index '{s}' has duplicate keys in existing data; duplicates not enforced for existing documents\n", .{ix.name});
}
}
@@ -1369,12 +1369,12 @@ test "ttl_sweep deletes expired documents and the deletion survives reopen" {
}
const coll = engine.get_collection("app", "sessions").?;
try testing.expectEqual(@as(usize, 6), coll.docs.count());
try testing.expectEqual(@as(usize, 6), coll.indexes.items[0].entries.items.len);
try testing.expectEqual(@as(usize, 6), coll.indexes.items[0].count());
// The cutoff is inclusive: doc 2 goes with doc 1.
try testing.expectEqual(@as(usize, 2), try engine.ttl_sweep(now_ms));
try testing.expectEqual(@as(usize, 4), coll.docs.count());
try testing.expectEqual(@as(usize, 4), coll.indexes.items[0].entries.items.len);
try testing.expectEqual(@as(usize, 4), coll.indexes.items[0].count());
try testing.expectEqual(@as(usize, 0), try index_count(gpa, &engine, "app", "sessions", "expireAt_1", .{ .datetime = now_ms - 120_000 }));
// The string and the missing field are untouched by any sweep.
try testing.expectEqual(@as(usize, 1), try index_count(gpa, &engine, "app", "sessions", "expireAt_1", .{ .string = "tomorrow" }));
@@ -1397,7 +1397,7 @@ test "ttl_sweep deletes expired documents and the deletion survives reopen" {
try testing.expectEqual(@as(usize, 3), coll.docs.count());
try testing.expectEqual(@as(usize, 1), coll.indexes.items.len);
try testing.expectEqual(@as(?i64, 60), coll.indexes.items[0].ttl);
try testing.expectEqual(@as(usize, 3), coll.indexes.items[0].entries.items.len);
try testing.expectEqual(@as(usize, 3), coll.indexes.items[0].count());
for ([_]i32{ 1, 2, 3 }) |id| {
const id_key = try bson.serialize_value(gpa, bson.Value{ .int32 = id });
defer gpa.free(id_key);

1161
src/index.zig
View File

File diff suppressed because it is too large Load Diff

74
src/spill.zig Normal file
View File

@@ -0,0 +1,74 @@
// Dev stress test for the overflow slab (records > 1024 bytes):
// zig run -O ReleaseFast src/spill.zig
// Keys straddling the spill threshold (10 B .. 100 KB) through insert,
// lookup, delete and iteration.
const std = @import("std");
const index = @import("index.zig");
const bson = @import("bson.zig");
fn doc_of(pairs: []const bson.Pair) bson.Document {
return .{ .arena = undefined, .pairs = pairs };
}
pub fn main() !void {
const gpa = std.heap.page_allocator;
var prng = std.Random.DefaultPrng.init(0x1234_5678);
const rand = prng.random();
var keys = [_]index.IndexKey{.{ .path = "tag", .descending = false }};
var ix = try index.Index.init(gpa, "tag", &keys, false, false, null);
// Keys straddling the spill threshold: inline, exactly at the limit,
// just over, and one very long. Each id is a short static string.
const lens = [_]usize{ 10, 1023, 1024, 1025, 2000, 100_000 };
var strings: [lens.len][]u8 = undefined;
var docs: [lens.len]bson.Document = undefined;
var pairs: [2]bson.Pair = undefined;
for (lens, 0..) |len, i| {
strings[i] = try gpa.alloc(u8, len);
for (strings[i]) |*c| c.* = 'a' + @as(u8, @intCast(rand.intRangeAtMost(u8, 0, 25)));
// add a distinguishing suffix so keys are unique
std.mem.copyForwards(u8, strings[i][len - 4 ..], &[_]u8{ @intCast(i), 0xff, 0x00, 0x00 });
pairs[0] = .{ .key = "_id", .value = .{ .int32 = @intCast(i) } };
pairs[1] = .{ .key = "tag", .value = .{ .string = strings[i] } };
docs[i] = doc_of(&pairs);
_ = try ix.add_doc(gpa, &docs[i], &[_]u8{ 'i', 'd', @intCast(i + 1) }, true);
}
std.debug.print("count={d} leaves={d} depth={d} overflow={d}\n", .{ ix.count(), ix.leaf_count, ix.depth, ix.overflow.items.len });
if (ix.overflow.items.len < 100_000) return error.NoSpill;
// Every entry is found by exact key.
for (lens, 0..) |_, i| {
var out: std.ArrayListUnmanaged([]const u8) = .empty;
defer out.deinit(gpa);
try ix.lookup_eq(gpa, &.{.{ .string = strings[i] }}, &out);
if (out.items.len != 1) {
std.debug.print("lookup {d} got {d}\n", .{ i, out.items.len });
return error.Bad;
}
if (!std.mem.eql(u8, out.items[0], &[_]u8{ 'i', 'd', @intCast(i + 1) })) return error.Bad;
}
// Delete the spilled ones and the inline ones alternately.
for (lens, 0..) |_, i| {
if (i % 2 == 0) continue;
ix.remove_doc(gpa, &docs[i], &[_]u8{ 'i', 'd', @intCast(i + 1) });
}
if (ix.count() != 3) return error.Bad;
for (lens, 0..) |_, i| {
if (i % 2 == 0) continue;
var out: std.ArrayListUnmanaged([]const u8) = .empty;
defer out.deinit(gpa);
try ix.lookup_eq(gpa, &.{.{ .string = strings[i] }}, &out);
if (out.items.len != 0) return error.Bad;
}
// Iteration still sees the survivors in order.
var it = ix.iter();
var seen: usize = 0;
while (it.next()) |e| {
seen += 1;
_ = e;
}
if (seen != 3) return error.Bad;
std.debug.print("SPILL OK (seen={d})\n", .{seen});
}

88
src/spill2.zig Normal file
View File

@@ -0,0 +1,88 @@
// Dev stress test: spilled records through leaf splits and internal levels.
// zig run -O ReleaseFast src/spill2.zig
// 5000 docs with 2 KiB keys: every record spills; the tree still answers
// exact lookups and deletes half of them.
const std = @import("std");
const index = @import("index.zig");
const bson = @import("bson.zig");
fn doc_of(pairs: []const bson.Pair) bson.Document {
return .{ .arena = undefined, .pairs = pairs };
}
pub fn main() !void {
const gpa = std.heap.page_allocator;
var keys = [_]index.IndexKey{.{ .path = "tag", .descending = false }};
var ix = try index.Index.init(gpa, "tag", &keys, false, false, null);
// 5000 docs, each with a 2 KiB key: spills on every record, forcing
// leaves and internal nodes to hold overflow references.
const N = 5000;
var ids: std.ArrayListUnmanaged([]u8) = .empty;
defer {
for (ids.items) |x| gpa.free(x);
ids.deinit(gpa);
}
var pairs: [2]bson.Pair = undefined;
var buf = try gpa.alloc(u8, 2000);
defer gpa.free(buf);
var facts: std.ArrayListUnmanaged(struct { key: []u8 }) = .empty;
defer {
for (facts.items) |f| gpa.free(f.key);
facts.deinit(gpa);
}
for (0..N) |i| {
for (buf) |*c| c.* = 'x';
// unique suffix
std.mem.writeInt(u32, buf[0..4], @intCast(i), .little);
const key = try gpa.dupe(u8, buf);
try facts.append(gpa, .{ .key = key });
const id = try std.fmt.allocPrint(gpa, "id{d}", .{i});
try ids.append(gpa, id);
pairs[0] = .{ .key = "_id", .value = .{ .int32 = @intCast(i) } };
pairs[1] = .{ .key = "tag", .value = .{ .string = key } };
const d = doc_of(&pairs);
_ = try ix.add_doc(gpa, &d, id, false);
}
std.debug.print("count={d} leaves={d} depth={d} overflow={d}\n", .{ ix.count(), ix.leaf_count, ix.depth, ix.overflow.items.len });
if (ix.count() != N) return error.Bad;
// Spot-check exact lookups.
var prng2 = std.Random.DefaultPrng.init(0xabc);
const rand2 = prng2.random();
for (0..300) |_| {
const i = rand2.intRangeAtMost(usize, 0, N - 1);
var out: std.ArrayListUnmanaged([]const u8) = .empty;
defer out.deinit(gpa);
try ix.lookup_eq(gpa, &.{.{ .string = facts.items[i].key }}, &out);
if (out.items.len != 1 or !std.mem.eql(u8, out.items[0], ids.items[i])) {
std.debug.print("lookup mismatch at {d}\n", .{i});
return error.Bad;
}
}
// Delete half (random), verify count and no leftover.
var order: std.ArrayListUnmanaged(usize) = .empty;
defer order.deinit(gpa);
for (0..N) |i| if (i % 2 == 0) try order.append(gpa, i);
rand2.shuffle(usize, order.items);
var removed: usize = 0;
for (order.items) |i| {
pairs[0] = .{ .key = "_id", .value = .{ .int32 = @intCast(i) } };
pairs[1] = .{ .key = "tag", .value = .{ .string = facts.items[i].key } };
const d = doc_of(&pairs);
ix.remove_doc(gpa, &d, ids.items[i]);
removed += 1;
if (ix.count() != N - removed) {
std.debug.print("count mismatch at {d}: {d} != {d}\n", .{ i, ix.count(), N - removed });
return error.Bad;
}
}
for (order.items) |i| {
var out: std.ArrayListUnmanaged([]const u8) = .empty;
defer out.deinit(gpa);
try ix.lookup_eq(gpa, &.{.{ .string = facts.items[i].key }}, &out);
if (out.items.len != 0) return error.Bad;
}
std.debug.print("SPILL2 OK (leaves={d} depth={d})\n", .{ ix.leaf_count, ix.depth });
}

174
src/stress.zig Normal file
View File

@@ -0,0 +1,174 @@
// Dev stress test for the index B+tree (not part of the build):
// zig run -O ReleaseFast src/stress.zig
// Bulk-builds 30k entries, inserts 20k more one at a time (splits at depth 2),
// deletes 16.6k randomly (empty-leaf cascades), drains everything, and checks
// lookups against a brute-force model throughout.
const std = @import("std");
const index = @import("index.zig");
const bson = @import("bson.zig");
fn doc_of(pairs: []const bson.Pair) bson.Document {
return .{ .arena = undefined, .pairs = pairs };
}
const Fact = struct { a: i32, b: i32 };
fn check_range(gpa: std.mem.Allocator, ix: *const index.Index, prefix: bson.Value, lo: ?bson.Value, hi: ?bson.Value, facts: []const Fact, alive: []const bool) !void {
var out: std.ArrayListUnmanaged([]const u8) = .empty;
defer out.deinit(gpa);
try ix.lookup_range(gpa, &.{prefix}, lo, true, hi, false, &out);
var expected: usize = 0;
for (facts, 0..) |f, fi| {
if (!alive[fi]) continue;
if (f.a != prefix.int32) continue;
if (lo) |l| if (f.b < l.int32) continue;
if (hi) |h| if (f.b >= h.int32) continue;
expected += 1;
}
if (out.items.len != expected) {
std.debug.print("MISMATCH: prefix={d} lo={?d} hi={?d}: got {d}, want {d}\n", .{ prefix.int32, if (lo) |l| l.int32 else null, if (hi) |h| h.int32 else null, out.items.len, expected });
std.process.exit(1);
}
}
pub fn main() !void {
const gpa = std.heap.page_allocator;
var prng = std.Random.DefaultPrng.init(0xBADCAFE);
const rand = prng.random();
const N = 30_000;
var ids: std.ArrayListUnmanaged([]u8) = .empty;
var facts: std.ArrayListUnmanaged(Fact) = .empty;
var alive: std.ArrayListUnmanaged(bool) = .empty;
var pairs: [2]bson.Pair = undefined;
// 1. Bulk build an index over N docs.
var keys = [_]index.IndexKey{ .{ .path = "a", .descending = false }, .{ .path = "b", .descending = false } };
var ix = try index.Index.init(gpa, "ab", &keys, false, false, null);
for (0..N) |i| {
const id = try std.fmt.allocPrint(gpa, "id{d}", .{i});
try ids.append(gpa, id);
const a = rand.intRangeAtMost(i32, 0, 99);
const b = rand.intRangeAtMost(i32, 0, 999);
try facts.append(gpa, .{ .a = a, .b = b });
try alive.append(gpa, true);
pairs[0] = .{ .key = "a", .value = .{ .int32 = a } };
pairs[1] = .{ .key = "b", .value = .{ .int32 = b } };
const d = doc_of(&pairs);
try ix.append_doc_entries(gpa, &d, id);
}
_ = try ix.finish_bulk(gpa, false);
std.debug.print("bulk: count={d} leaves={d} depth={d} nodes={d}\n", .{ ix.count(), ix.leaf_count, ix.depth, ix.nodes.items.len });
if (ix.count() != N) return error.BadCount;
// Random range checks against brute force.
for (0..500) |_| {
const a = rand.intRangeAtMost(i32, 0, 99);
const lo_v = rand.intRangeAtMost(i32, -10, 1009);
const hi_v = rand.intRangeAtMost(i32, -10, 1009);
const use_lo = rand.boolean();
const use_hi = rand.boolean();
try check_range(gpa, &ix, .{ .int32 = a }, if (use_lo) .{ .int32 = lo_v } else null, if (use_hi) .{ .int32 = hi_v } else null, facts.items, alive.items);
}
// 2. Incremental inserts, random order (splits + rebalancing-free path).
const M = 20_000;
for (0..M) |i| {
const id = try std.fmt.allocPrint(gpa, "new{d}", .{i});
try ids.append(gpa, id);
const a = rand.intRangeAtMost(i32, 0, 99);
const b = rand.intRangeAtMost(i32, 0, 999);
try facts.append(gpa, .{ .a = a, .b = b });
try alive.append(gpa, true);
pairs[0] = .{ .key = "a", .value = .{ .int32 = a } };
pairs[1] = .{ .key = "b", .value = .{ .int32 = b } };
const d = doc_of(&pairs);
_ = try ix.add_doc(gpa, &d, id, false);
}
std.debug.print("after inserts: count={d} leaves={d} depth={d} nodes={d}\n", .{ ix.count(), ix.leaf_count, ix.depth, ix.nodes.items.len });
if (ix.count() != N + M) return error.BadCount;
for (0..500) |_| {
const a = rand.intRangeAtMost(i32, 0, 99);
const lo_v = rand.intRangeAtMost(i32, -10, 1009);
const hi_v = rand.intRangeAtMost(i32, -10, 1009);
const use_lo = rand.boolean();
const use_hi = rand.boolean();
try check_range(gpa, &ix, .{ .int32 = a }, if (use_lo) .{ .int32 = lo_v } else null, if (use_hi) .{ .int32 = hi_v } else null, facts.items, alive.items);
}
// 3. Delete every 3rd doc in random order (empty-leaf cascades,
// one-child internals).
var order: std.ArrayListUnmanaged(usize) = .empty;
for (0..N + M) |i| if (i % 3 == 0) try order.append(gpa, i);
rand.shuffle(usize, order.items);
for (order.items) |i| {
const a = facts.items[i].a;
const b = facts.items[i].b;
pairs[0] = .{ .key = "a", .value = .{ .int32 = a } };
pairs[1] = .{ .key = "b", .value = .{ .int32 = b } };
const d = doc_of(&pairs);
ix.remove_doc(gpa, &d, ids.items[i]);
alive.items[i] = false;
}
std.debug.print("after deletes: count={d} leaves={d} depth={d} nodes={d}\n", .{ ix.count(), ix.leaf_count, ix.depth, ix.nodes.items.len });
if (ix.count() != (N + M) - (N + M) / 3 - 1) return error.BadCount;
for (0..500) |_| {
const a = rand.intRangeAtMost(i32, 0, 99);
const lo_v = rand.intRangeAtMost(i32, -10, 1009);
const hi_v = rand.intRangeAtMost(i32, -10, 1009);
const use_lo = rand.boolean();
const use_hi = rand.boolean();
try check_range(gpa, &ix, .{ .int32 = a }, if (use_lo) .{ .int32 = lo_v } else null, if (use_hi) .{ .int32 = hi_v } else null, facts.items, alive.items);
}
// 4. Equality lookups still exact.
for (0..300) |_| {
const a = rand.intRangeAtMost(i32, 0, 99);
var out: std.ArrayListUnmanaged([]const u8) = .empty;
defer out.deinit(gpa);
try ix.lookup_eq(gpa, &.{.{ .int32 = a }}, &out);
var expected: usize = 0;
for (facts.items, 0..) |f, fi| { if (alive.items[fi] and f.a == a) expected += 1; }
if (out.items.len != expected) {
std.debug.print("EQ MISMATCH a={d}: got {d} want {d}\n", .{ a, out.items.len, expected });
return error.BadCount;
}
}
// 5. Delete everything (empty-leaf cascades, one-child internals),
// then verify the tree still works for fresh inserts.
var live: std.ArrayListUnmanaged(usize) = .empty;
defer live.deinit(gpa);
for (facts.items, 0..) |_, i| if (alive.items[i]) try live.append(gpa, i);
rand.shuffle(usize, live.items);
var remaining = live.items.len;
for (live.items) |i| {
const a = facts.items[i].a;
const b = facts.items[i].b;
pairs[0] = .{ .key = "a", .value = .{ .int32 = a } };
pairs[1] = .{ .key = "b", .value = .{ .int32 = b } };
const d = doc_of(&pairs);
ix.remove_doc(gpa, &d, ids.items[i]);
remaining -= 1;
if (ix.count() != remaining) {
std.debug.print("COUNT MISMATCH during drain: {d} != {d}\n", .{ ix.count(), remaining });
return error.BadCount;
}
}
std.debug.print("after full drain: count={d} leaves={d} depth={d} nodes={d}\n", .{ ix.count(), ix.leaf_count, ix.depth, ix.nodes.items.len });
if (ix.count() != 0) return error.BadCount;
// The drained tree still accepts and finds entries.
pairs[0] = .{ .key = "a", .value = .{ .int32 = 7 } };
pairs[1] = .{ .key = "b", .value = .{ .int32 = 42 } };
const d2 = doc_of(&pairs);
_ = try ix.add_doc(gpa, &d2, "final", false);
var out: std.ArrayListUnmanaged([]const u8) = .empty;
defer out.deinit(gpa);
try ix.lookup_eq(gpa, &.{.{ .int32 = 7 }}, &out);
if (out.items.len != 1) return error.BadCount;
var it = ix.iter();
if (it.next() == null) return error.BadCount;
if (it.next() != null) return error.BadCount;
std.debug.print("STRESS OK\n", .{});
}

38
tests/e2e/results/phase2.txt Normal file
View File

@@ -0,0 +1,38 @@
# Phase 2 gate — mongo-lite vs MongoDB 8.3.7, 1g dataset / ~16k docs
# Ratio < 1.0 = mongo-lite faster. Reproduce: bash tests/e2e/compare-run.sh 1g 16k
# This run includes roadmap item 1 (B+tree over the encoded keys).
# Compare: tests/e2e/results/phase1.txt (pre-tree baseline).
benchmark mongo-lite mongodb ratio
insertOne (sequential) ×200 0.19 ms 4.1 ms 0.0x
bulk insert throughput 810.4 MB/s 714.4 MB/s 1.1x
docs loaded 65,536 65,536 1.0x
createIndex({k: 1}) 50.8 ms 82.4 ms 0.6x
countDocuments({}) 1.5 ms 13.8 ms 0.1x
findOne({_id: <ObjectId>}) 0.57 ms 0.67 ms 0.9x
findOne({k: 500}) (indexed) 0.57 ms 1.8 ms 0.3x
find({p: {$gte,$lt}}).count() (scan) 20.3 ms 12.9 ms 1.6x
find({}).sort({_id:-1}).limit(20) 6.2 ms 2.7 ms 2.3x
find({}, {proj}).limit(1000) 3.7 ms 4.5 ms 0.8x
aggregate $group by k 11.5 ms 15.5 ms 0.7x
updateOne({_id}) ×50 0.17 ms 0.19 ms 0.9x
updateMany({k: 7}, {$inc}) 1.8 ms 6.7 ms 0.3x
deleteOne({_id}) + insertOne 0.50 ms 5.0 ms 0.1x
node client RSS 152 MB 156 MB 1.0x
server RSS 1974 MB 1474 MB
kill -9 reopen 0.8s 1.3s
db on disk 1028MB 96MB
# Item 1 (B+tree over encoded keys) deltas vs phase1:
# updateMany 17.3 -> 1.8 ms (2.8x slower than mongod -> 3.7x faster):
# entry removal was a per-entry binary search into a sorted
# array with an orderedRemove memmove behind it; now it is a
# descent plus a leaf-local slot removal.
# createIndex 62.4 -> 50.8 ms (bulk packing replaces append+sort)
# findOne(k:500) indexed 0.64 -> 0.57 ms (unchanged shape, tree search)
#
# Remaining gaps and where they are addressed:
# db on disk 11x -> Phase 3 (block-compressed log)
# sort+limit 2.3x -> Phase 2 (ordered _id index)
# range-scan 1.6x -> Phase 4 (contiguous byte storage, not the matcher)
# server RSS 1.3x -> Phase 4 (per-document arena -> byte storage)