index/commands: bulk index build, binary-searched ranges, limit push-down
createIndex built the entry array one document at a time, and each
insert kept the array sorted by memmoving the tail -- O(n^2) bytes moved
over a full build, which was the entire cost of the operation. Entries
are now appended unsorted and ordered once (append_doc_entries +
finish_bulk), with uniqueness checked by a single adjacent-pair scan
instead of a binary search per document. build_all_indexes, which runs
for every index on every open, takes the same path.
createIndex over 65,536 documents, measured A/B:
{k: 1} 649ms -> 56ms
{s: 1} unique 678ms -> 53ms
{p: 1, k: -1} 653ms -> 54ms
lookup_range binary-searched only the equality prefix and then scanned
that whole band applying a filter, so a range on the first component of
an index touched every entry in it. Both ends are now binary searches
over the component the array is already sorted on, clamped into the
equality band. Note this does not move the range-scan row in compare.js:
that query filters on p, which has no index there, so it is a collection
scan and belongs to the matcher.
cmd_find passed a hardcoded 0 as the scan limit, so find().limit(n)
materialized the entire collection before slicing. It now stops once the
page is filled, when there is no sort to order the matches first; the
bound covers the skipped prefix because the scan counts matches rather
than returned documents.
lookup_range's bounds are checked by a new randomized test that compares
the result count against a brute-force filter over 600 generated
queries, with values chosen from a small domain so equal keys and the
inclusive/exclusive edges come up constantly. Verified it fails when
either bound is swapped.
This commit is contained in:
@@ -564,7 +564,15 @@ fn cmd_find(ctx: *Context, msg: *wire.Message, reply: *wire.Reply) !void {
|
||||
|
||||
var matched: std.ArrayListUnmanaged(*const bson.Document) = .empty;
|
||||
defer matched.deinit(ctx.gpa);
|
||||
_ = try scan_matching(ctx, db_name, coll_name, filter, 0, &matched);
|
||||
// Stop scanning once the page is filled. Only sound without a sort,
|
||||
// which has to see every match before it can tell which ones the page
|
||||
// contains, and the bound has to cover the skipped prefix too because
|
||||
// scan_matching counts matches rather than returned documents.
|
||||
const need: usize = if (sort_keys.len > 0 or limit == 0) 0 else blk: {
|
||||
const skip_usize = std.math.cast(usize, skip) orelse break :blk 0;
|
||||
break :blk skip_usize +| limit;
|
||||
};
|
||||
_ = try scan_matching(ctx, db_name, coll_name, filter, need, &matched);
|
||||
|
||||
if (sort_keys.len > 0) {
|
||||
try query.sort_docs(reply.arena_alloc(), matched.items, sort_keys);
|
||||
|
||||
Reference in New Issue
Block a user