dev/MultiforaDB
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

index/commands: stream whole-index scans; add a reverse leaf iterator

Browse Source 
A whole-index read used to materialize every candidate before the caller saw
the first one. At the tens-of-GB target that is a list of every offset in the
collection -- ~160 MB for a countDocuments({}) over 20 million documents --
which defeats the point of moving storage to disk. Cursors are M1, but
*streaming a scan* has to exist now.

`Candidates` is the one loop candidates arrive through, whatever produced them:
a plan's materialized lookups, or the index read end to end. That keeps this
file's governing invariant -- an index only generates candidates, the full
filter is re-applied to every one -- in a single place. A narrowed plan still
materializes, because its multikey/$in dedupe genuinely needs the whole set and
is bounded by selectivity.

`RevIter` walks `Node.prev`, which has always been maintained and which nothing
had ever read: a descending scan materialized the whole index and reversed the
list. `find({}).sort({_id:-1}).limit(20)` becomes O(20).

The unfiltered fallback now walks the _id_ index instead of the docs map. That
is ordered rather than hash-ordered, and it does not depend on a structure that
is about to be deleted.

`Plan.full_scan()` refuses multikey indexes, since one document contributes
several entries there and a stream cannot dedupe what `search` did. The check is
currently redundant -- the planner refuses to order a multikey index anyway --
and is kept because the two guards protect different things. Stated precisely
in both places after checking: the commands.zig test reddens only when *both*
guards are removed, which is what that test actually pins.

--

This also broke e2e6's compaction check, and the fix there is the more
interesting half.

The check required peak/final > 1.4 and got 1.28. The final size was identical
to the byte (2,398,065 vs 2,398,064) -- compaction reclaimed exactly as before
-- and only the peak moved. Isolated to one variable: changing just the order
updateMany({}) walks its matches moves peak/final between 1.65 and 1.28, because
compaction can also fire from the once-per-second TTL monitor and whether one
lands inside the batch shifts the peak a long way while leaving the outcome
unchanged. The threshold was measuring the schedule.

Replaced with `peak > final`, which measures the shape instead: an append-only
log grows monotonically, so its maximum *is* its final size, and a file that
was ever larger than it ended can only have been rewritten.

Worth recording why the obvious alternative does not work. An absolute size
bound cannot distinguish a working compactor here: the payload is one repeated
character, so ~48 MB of records LZ4-compress to ~3 MB whether or not anything is
reclaimed -- with compaction disabled entirely the file still ends at 3.1 MB. I
first wrote the comment claiming that bound was the strong one, then measured it
and found the opposite; `peak > final` is what goes red.
This commit is contained in:
Alexey Shakhmatov
2026-08-03 20:05:38 +03:00
parent 491a4d0a6a
commit 9390021b1e
3 changed files with 321 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files

201
src/index.zig
View File

@@ -706,6 +706,55 @@ pub const Index = struct {
return .{ .ix = self, .leaf = self.first_leaf, .slot = 0 };
}
/// Reverse ordered iteration. Leaves are doubly linked and `prev` has
/// always been maintained -- nothing walked it until now, so a descending
/// scan had to materialize every candidate and reverse the list. This turns
/// `find({}).sort({_id: -1}).limit(20)` from O(collection) into O(20).
pub const RevIter = struct {
ix: *const Index,
leaf: u32,
/// One past the slot to yield next, so 0 means this leaf is done.
slot: u32,
pub fn next(self: *RevIter) ?EntryRef {
const ix = self.ix;
while (self.leaf != 0) {
if (self.slot > 0) {
self.slot -= 1;
return .{
.key = ix.key_of(self.leaf, self.slot),
.off = ix.off_of(self.leaf, self.slot),
};
}
const prev = ix.page(self.leaf).prev;
self.leaf = prev;
if (prev != 0) self.slot = ix.page(prev).count;
}
return null;
}
};
pub fn iter_reverse(self: *const Index) RevIter {
const last = self.descend_last();
return .{ .ix = self, .leaf = last, .slot = self.page(last).count };
}
/// The rightmost leaf. An internal node's children are `first_child`
/// followed by one per separator, so the last child is the last slot's
/// `extra` (or `first_child` when the node holds no separators -- which
/// removal can leave behind, since it never rebalances).
fn descend_last(self: *const Index) u32 {
var node_id = self.root;
while (self.page(node_id).is_leaf == 0) {
const node = self.page(node_id);
node_id = if (node.count == 0)
node.first_child
else
get_slot(node, node.count - 1).extra;
}
return node_id;
}
/// Ordered iteration starting at the first entry whose key is not less
/// than `prefix` (prefix semantics). Used by the equality/range
/// searches and the TTL sweep's datetime band.
@@ -1681,6 +1730,33 @@ fn default_name(gpa: std.mem.Allocator, key_pairs: []const bson.Pair) ![]u8 {
/// independent, and sorting handles direction itself.
/// Entry ordering. No tie-break beyond the key: entries no longer carry a
/// document, and within one document's batch equal keys are deduped anyway.
/// A stream of candidate slab offsets, however they were produced: a plan's
/// materialized lookups, or the index read end to end. The point is that the
/// consumer is one loop, so the governing invariant of this file -- an index
/// only generates candidates, the full filter is re-applied to every one --
/// lives in exactly one place regardless of which shape produced them.
///
/// A `.scan`/`.scan_rev` holds an iterator positioned in the tree, so it must
/// be drained (or dropped) before this collection is written to. The offsets it
/// yields are values and are unaffected by any later mutation.
pub const Candidates = union(enum) {
scan: Index.Iter,
scan_rev: Index.RevIter,
list: struct { items: []const u64, i: usize = 0 },
pub fn next(self: *Candidates) ?u64 {
switch (self.*) {
.scan => |*it| return if (it.next()) |e| e.off else null,
.scan_rev => |*it| return if (it.next()) |e| e.off else null,
.list => |*l| {
if (l.i >= l.items.len) return null;
defer l.i += 1;
return l.items[l.i];
},
}
}
};
pub fn compare_entries(a: Entry, b: Entry) std.math.Order {
return std.mem.order(u8, a.key, b.key);
}
@@ -1839,6 +1915,30 @@ pub const Plan = struct {
/// That order is the reverse of the index's.
backward: bool = false,
/// Whether this plan reads the whole index with no narrowing, which is
/// the case a streaming leaf walk can answer without materializing
/// anything. At the M0 scale that matters: a `countDocuments({})` over
/// 20 million documents would otherwise build a 160 MB list of offsets
/// before the caller sees the first one.
///
/// Multikey indexes are excluded. One document contributes several entries
/// there, so a full walk yields it more than once -- `search` dedupes, a
/// stream cannot.
///
/// That check is currently redundant: `index_provides_sort` refuses
/// multikey, and a run-0 plan with no range is only formed when it supplies
/// the sort, so no multikey plan reaches here today. Kept deliberately --
/// the two guards protect different things, and if the planner ever learns
/// to order a multikey index, streaming must not silently start returning
/// duplicates. Verified by removing both: the commands.zig test
/// "a sorted full scan over a multikey index returns each document once"
/// goes red only then, which is the honest statement of what pins this.
pub fn full_scan(self: *const Plan) bool {
if (self.index.multikey) return false;
if (self.lo != null or self.hi != null) return false;
return self.lookup_keys.items.len == 1 and self.lookup_keys.items[0].len == 0;
}
pub fn deinit(self: *Plan, gpa: std.mem.Allocator) void {
for (self.lookup_keys.items) |k| gpa.free(k);
self.lookup_keys.deinit(gpa);
@@ -2272,6 +2372,107 @@ test "unique conflict across documents, replace of own entries allowed" {
try expect_offs(gpa, &ix, &.{.{ .int32 = 10 }}, &.{});
}
test "iter_reverse yields every entry in exact reverse order" {
// Node.prev has always been maintained and nothing walked it, so a
// descending scan materialized the whole index and reversed the list. The
// interesting cases are structural, not arithmetic: enough entries to build
// several leaves and an interior level, so descend_last has to follow the
// last separator's child rather than first_child, and a run of removals
// afterwards because removal never rebalances -- it can leave an internal
// node with no separators at all, which is the branch descend_last would
// otherwise get wrong.
//
// Mutation checks: start iter_reverse at `first_leaf` and the order is
// wrong from the first entry; make descend_last follow `first_child`
// unconditionally and it silently misses everything to the right.
const gpa = testing.allocator;
var ix = try simple_index(gpa, &.{"a"}, false, false);
defer ix.deinit(gpa);
const n = 400;
for (0..n) |i| {
const d = try bytes_of(gpa, &.{
.{ .key = "_id", .value = .{ .int32 = @intCast(i + 1) } },
.{ .key = "a", .value = .{ .int32 = @intCast(i + 1) } },
});
defer gpa.free(d);
_ = try ix.add_doc(gpa, d, @intCast(i + 1), false);
}
try testing.expect(ix.depth >= 1); // an interior level exists
var fwd: std.ArrayListUnmanaged(u64) = .empty;
defer fwd.deinit(gpa);
var it = ix.iter();
while (it.next()) |e| try fwd.append(gpa, e.off);
try testing.expectEqual(@as(usize, n), fwd.items.len);
var rev: std.ArrayListUnmanaged(u64) = .empty;
defer rev.deinit(gpa);
var rit = ix.iter_reverse();
while (rit.next()) |e| try rev.append(gpa, e.off);
try testing.expectEqual(fwd.items.len, rev.items.len);
for (fwd.items, 0..) |off, i| {
try testing.expectEqual(off, rev.items[rev.items.len - 1 - i]);
}
// Now churn: delete most of it and re-check, since removal reshapes the
// interior without rebalancing.
for (0..n) |i| {
if (i % 4 == 0) continue;
ix.remove_off(@intCast(i + 1));
}
fwd.clearRetainingCapacity();
rev.clearRetainingCapacity();
var it2 = ix.iter();
while (it2.next()) |e| try fwd.append(gpa, e.off);
var rit2 = ix.iter_reverse();
while (rit2.next()) |e| try rev.append(gpa, e.off);
try testing.expectEqual(fwd.items.len, rev.items.len);
for (fwd.items, 0..) |off, i| {
try testing.expectEqual(off, rev.items[rev.items.len - 1 - i]);
}
}
test "Candidates streams the same offsets a materialized plan would" {
// The streaming path and the materializing path must be interchangeable,
// because scan_sorted picks between them on a property of the plan. Compare
// them directly rather than trusting that.
//
// The multikey hazard full_scan() guards against lives in commands.zig's
// "a sorted full scan over a multikey index returns each document once",
// not here -- this index is not multikey. See Plan.full_scan for why that
// test only reddens when *both* multikey guards are removed.
const gpa = testing.allocator;
var ix = try simple_index(gpa, &.{"a"}, false, false);
defer ix.deinit(gpa);
for (0..50) |i| {
const d = try bytes_of(gpa, &.{
.{ .key = "_id", .value = .{ .int32 = @intCast(i + 1) } },
.{ .key = "a", .value = .{ .int32 = @intCast(i % 7) } },
});
defer gpa.free(d);
_ = try ix.add_doc(gpa, d, @intCast(i + 1), false);
}
var streamed: std.ArrayListUnmanaged(u64) = .empty;
defer streamed.deinit(gpa);
var c: Candidates = .{ .scan = ix.iter() };
while (c.next()) |off| try streamed.append(gpa, off);
var listed: std.ArrayListUnmanaged(u64) = .empty;
defer listed.deinit(gpa);
try ix.lookup_eq(gpa, &.{}, &listed);
var c2: Candidates = .{ .list = .{ .items = listed.items } };
var from_list: std.ArrayListUnmanaged(u64) = .empty;
defer from_list.deinit(gpa);
while (c2.next()) |off| try from_list.append(gpa, off);
try testing.expectEqualSlices(u64, listed.items, streamed.items);
try testing.expectEqualSlices(u64, listed.items, from_list.items);
}
test "lookup_exact matches whole keys only" {
// The point of exact byte equality rather than cmp_prefix: this answers
// "is this document present", which is what the engine will ask once the