index/commands: stream whole-index scans; add a reverse leaf iterator

A whole-index read used to materialize every candidate before the caller saw
the first one. At the tens-of-GB target that is a list of every offset in the
collection -- ~160 MB for a countDocuments({}) over 20 million documents --
which defeats the point of moving storage to disk. Cursors are M1, but
*streaming a scan* has to exist now.

`Candidates` is the one loop candidates arrive through, whatever produced them:
a plan's materialized lookups, or the index read end to end. That keeps this
file's governing invariant -- an index only generates candidates, the full
filter is re-applied to every one -- in a single place. A narrowed plan still
materializes, because its multikey/$in dedupe genuinely needs the whole set and
is bounded by selectivity.

`RevIter` walks `Node.prev`, which has always been maintained and which nothing
had ever read: a descending scan materialized the whole index and reversed the
list. `find({}).sort({_id:-1}).limit(20)` becomes O(20).

The unfiltered fallback now walks the _id_ index instead of the docs map. That
is ordered rather than hash-ordered, and it does not depend on a structure that
is about to be deleted.

`Plan.full_scan()` refuses multikey indexes, since one document contributes
several entries there and a stream cannot dedupe what `search` did. The check is
currently redundant -- the planner refuses to order a multikey index anyway --
and is kept because the two guards protect different things. Stated precisely
in both places after checking: the commands.zig test reddens only when *both*
guards are removed, which is what that test actually pins.

--

This also broke e2e6's compaction check, and the fix there is the more
interesting half.

The check required peak/final > 1.4 and got 1.28. The final size was identical
to the byte (2,398,065 vs 2,398,064) -- compaction reclaimed exactly as before
-- and only the peak moved. Isolated to one variable: changing just the order
updateMany({}) walks its matches moves peak/final between 1.65 and 1.28, because
compaction can also fire from the once-per-second TTL monitor and whether one
lands inside the batch shifts the peak a long way while leaving the outcome
unchanged. The threshold was measuring the schedule.

Replaced with `peak > final`, which measures the shape instead: an append-only
log grows monotonically, so its maximum *is* its final size, and a file that
was ever larger than it ended can only have been rewritten.

Worth recording why the obvious alternative does not work. An absolute size
bound cannot distinguish a working compactor here: the payload is one repeated
character, so ~48 MB of records LZ4-compress to ~3 MB whether or not anything is
reclaimed -- with compaction disabled entirely the file still ends at 3.1 MB. I
first wrote the comment claiming that bound was the strong one, then measured it
and found the opposite; `peak > final` is what goes red.
This commit is contained in:
2026-08-03 20:05:38 +03:00
parent 491a4d0a6a
commit 9390021b1e
3 changed files with 321 additions and 23 deletions

View File

@@ -288,10 +288,26 @@ async function phase2(client, { users, aliceId }) {
// reclaimed it). ~48 MB of records are written here and ~12 MB of it
// survives, so without compaction the file would end near 48 MB.
//
// Both bounds are relative to the live size on purpose: compaction now
// triggers on the share of the log that is garbage rather than on bytes
// appended, so the absolute peak depends on when that share crosses the
// threshold and is not a stable number to assert on.
// `peak > final` is the assertion with teeth, and it is worth saying why,
// because the intuitive alternative does not work here. The log is
// LZ4-compressed and this payload is one repeated character, so ~48 MB of
// records compress to ~3 MB whether or not anything is reclaimed -- with
// compaction disabled entirely the file still ends at only 3.1 MB. So no
// absolute size bound distinguishes a working compactor from a broken one at
// this scale; the 24 MB below is a sanity guard, nothing more.
//
// What does distinguish them is the shape: an append-only log grows
// monotonically, so its maximum *is* its final size. A file that was ever
// larger than it ended can only have been rewritten. Verified by disabling
// compaction: final 3.1 MB against a peak of 2.7 MB, and this check goes red.
//
// The previous form required peak > final * 1.4, which measured the schedule
// rather than the engine: compaction also fires from the once-per-second TTL
// monitor, so whether one lands inside this batch moves the peak a long way
// while leaving the result identical. Measured by changing only the order
// updateMany({}) walks its matches -- hash order gave 1.65, _id order 1.28,
// and the final sizes differed by one byte. Any threshold between those two
// fails for a reason that has nothing to do with compaction.
let peakSize = fs.statSync(DBFILE).size;
const watcher = setInterval(() => {
const s = fs.statSync(DBFILE).size;
@@ -305,6 +321,9 @@ async function phase2(client, { users, aliceId }) {
clearInterval(watcher);
const logSize = fs.statSync(DBFILE).size;
if (process.env.E2E6_DEBUG || process.env.E2E6_PEAK) {
console.log(`DEBUG phase2 peak=${peakSize} final=${logSize} ratio=${(peakSize / logSize).toFixed(2)}`);
}
if (process.env.E2E6_DEBUG) {
const dbg = await users.findOne({ name: 'alice' });
console.log('DEBUG phase2 after replace+delete alice _id:', String(dbg._id), 'same:', String(dbg._id) === String(aliceId));
@@ -312,7 +331,7 @@ async function phase2(client, { users, aliceId }) {
}
check(
'compaction reclaimed junk (file ~ live size)',
logSize < 24 * 1024 * 1024 && peakSize > logSize * 1.4,
logSize < 24 * 1024 * 1024 && peakSize > logSize,
`file peaked at ${(peakSize / 1e6).toFixed(1)}MB, ended at ${(logSize / 1e6).toFixed(1)}MB after ~48MB of records were written (~12MB live)`,
);
check('bulk survivors intact', (await bulk.findOne({ _id: 1999 })).payload === payload2);