index/commands: bulk index build, binary-searched ranges, limit push-down

createIndex built the entry array one document at a time, and each
insert kept the array sorted by memmoving the tail -- O(n^2) bytes moved
over a full build, which was the entire cost of the operation. Entries
are now appended unsorted and ordered once (append_doc_entries +
finish_bulk), with uniqueness checked by a single adjacent-pair scan
instead of a binary search per document. build_all_indexes, which runs
for every index on every open, takes the same path.

  createIndex over 65,536 documents, measured A/B:
    {k: 1}          649ms -> 56ms
    {s: 1} unique   678ms -> 53ms
    {p: 1, k: -1}   653ms -> 54ms

lookup_range binary-searched only the equality prefix and then scanned
that whole band applying a filter, so a range on the first component of
an index touched every entry in it. Both ends are now binary searches
over the component the array is already sorted on, clamped into the
equality band. Note this does not move the range-scan row in compare.js:
that query filters on p, which has no index there, so it is a collection
scan and belongs to the matcher.

cmd_find passed a hardcoded 0 as the scan limit, so find().limit(n)
materialized the entire collection before slicing. It now stops once the
page is filled, when there is no sort to order the matches first; the
bound covers the skipped prefix because the scan counts matches rather
than returned documents.

lookup_range's bounds are checked by a new randomized test that compares
the result count against a brute-force filter over 600 generated
queries, with values chosen from a small domain so equal keys and the
inclusive/exclusive edges come up constantly. Verified it fails when
either bound is swapped.
This commit is contained in:
2026-08-02 18:27:28 +03:00
parent 556ad7dc86
commit ff37e6c813
3 changed files with 183 additions and 25 deletions

View File

@@ -352,15 +352,17 @@ pub const Engine = struct {
return error.IndexOptionsConflict;
}
// Build entries over the existing documents, checking uniqueness as
// we go (the index is not exposed until the end, so mutating it is
// safe). Each document's batch is inserted immediately, so on any
// later failure the deferred ix.deinit frees every inserted entry
// key. Nothing is persisted on failure.
// Build entries over the existing documents (the index is not
// exposed until the end, so mutating it is safe). Entries are
// appended unsorted and ordered once at the end — inserting each
// document into a sorted array memmoves the tail every time, which
// is what made this quadratic. On any failure the deferred
// ix.deinit frees every appended key. Nothing is persisted.
var doc_it = coll.docs.iterator();
while (doc_it.next()) |entry| {
_ = try ix.add_doc(self.gpa, entry.value_ptr.*, entry.key_ptr.*, true);
try ix.append_doc_entries(self.gpa, entry.value_ptr.*, entry.key_ptr.*);
}
_ = try ix.finish_bulk(true);
// Reserve the collection slot, then persist and publish.
try coll.indexes.ensureUnusedCapacity(self.gpa, 1);
@@ -619,16 +621,17 @@ pub const Engine = struct {
if (ix.entries.items.len > 0) continue; // defensive
var doc_it = coll_entry.value_ptr.docs.iterator();
while (doc_it.next()) |doc_entry| {
const duplicate = ix.add_doc(self.gpa, doc_entry.value_ptr.*, doc_entry.key_ptr.*, false) catch |err| switch (err) {
ix.append_doc_entries(self.gpa, doc_entry.value_ptr.*, doc_entry.key_ptr.*) catch |err| switch (err) {
error.ParallelArrays => {
std.debug.print("mongo-lite: WARNING: index '{s}' cannot index an existing document; entry skipped\n", .{ix.name});
continue;
},
else => return err,
};
if (duplicate) {
std.debug.print("mongo-lite: WARNING: unique index '{s}' has duplicate keys in existing data; duplicates not enforced for existing documents\n", .{ix.name});
}
}
// Tolerated, not enforced: the database must always open.
if (try ix.finish_bulk(false)) {
std.debug.print("mongo-lite: WARNING: unique index '{s}' has duplicate keys in existing data; duplicates not enforced for existing documents\n", .{ix.name});
}
}
}