dev/MultiforaDB
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

db/pager/tests: cleanup pass over the free list

Browse Source 
No behaviour is meant to change and the gate confirms it: 1.94x / 679.2 MB
reclaimed / 9 rebuilds and 2.46x / 934.0 MB / 13 on the two 16 KiB lines,
0.0 MB on the 200-byte line, all identical to the numbers recorded for them.

Deduplication. `pages_for` was written in db.zig and again in pager.zig and
twice more inline; there is now one, public, and the two pre-existing copies
call it. `pages_per_map_align` replaces three hand-rolled `map_align /
page_size`. `SlabRun.window_first` was a stored field that could never legally
disagree with `first` and was maintained by hand at two sites -- now a method.
`keep_piece` re-derived `SlabRun.window_count` character for character; it
calls it. `insert_run` scanned linearly for a position `run_of` binary-searches
for, which made loading a fragmented catalog quadratic; both now go through one
`run_lower_bound`. Freeing a run's window map was written four times; one
helper. The 20% rebuild share was stated in `note_compact` and again in
`wants_rebuild`, with a comment arguing at length that they must be the same
number -- `worth_rewriting` makes that structural.

Efficiency. The identity assert in `reclaim_windows` called `dead_located()`,
an O(every window) walk, and `assert_msg` is live in ReleaseFast -- so it
doubled the scan the reclamation was about to make (2.75 MB streamed twice per
reclaiming checkpoint at the 21 GB the gate targets). `dead_located` is now a
maintained counter, the check is O(1) in every build, and the scan cross-checks
it while it is there. `SlabRun.full` lets a run with nothing to give be copied
without its counters being read at all, so the common case is O(runs) rather
than O(windows).

The pager's two allocation policies were hand-copying the claim step, and the
copy had already lost two of the three preconditions -- `alloc_slab_run` never
checked `pages <= reserved_pages`. Both now go through `claim_locked`.

`reclaimed_bytes` moves from Collection to Engine, beside `compactions`, which
is how it is read and the only place it can be honest: a life-of-the-process
total must not lose a dropped collection's share. Both join `Counters`, so
`slab_stats` stops opening `counter_lock` by hand.

The ownership assertion in `write_catalog` was gated on `is_test or Debug`, a
predicate nothing else in the codebase uses, which left the one silent failure
this design can produce unchecked in ReleaseSafe. It is now `!= ReleaseFast`,
the line `protect_stable` already draws. Measured: no change to the suite's
runtime.

Altitude. `note_checkpoint` was called from exactly one place, the tail of
`upsert` -- so a delete armed no checkpoint by any route, which is why
reclamation only ever ran when the *rebuild* trigger fired and the rebuild then
reset the window map it would have used. `remove` and the TTL sweep arm one
now, next to the `note_compact` calls that were added for the same omission a
milestone ago. `compact`'s leading checkpoint stays, demoted in its comment
from the mechanism to the local ordering it actually guarantees.

serverStatus reports `allocTailBytes`/`freeReadyBytes` instead of page counts,
so the harness stops hard-coding 4096 -- the kind of constant this milestone
was blindsided by once already.

tests/e2e/churn.js: `deleteMany({_id: {$in: [5000 ids]}})` exceeded
`index.max_combos`, so the planner refused the index and every delete became a
full collection scan re-filtering each document against 5000 members. That was
the entire runtime of the harness. One delete spec per id instead: the 40k x
16 KiB gate goes 57 s -> 6 s, and the 150k x 200 B line 483 s -> 3 s, with
identical output. Also: the per-round `countDocuments` is gone (the harness
knows the count), and the server log is a bounded ring rather than a rope that
grows with everything the server ever said.

Reverted from the review: reusing one MongoClient across the startup poll. A
client whose first connect fails tears its topology down and every later
command on it fails identically, so it turns "not up yet" into "never comes
up" -- it broke the first run. The reason is now a comment.

187/187 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz, e2e 49, e2e2
concurrent 2 and the crash pair, e2e3 16, e2e4 17, e2e6 72, e2e7 86,
crash-fuzz 60 cycles.
This commit was merged in pull request #1.
This commit is contained in:
A.Shakhmatov
2026-08-09 19:08:15 +03:00
parent 36311d0faa
commit a748a3d08c
6 changed files with 333 additions and 215 deletions
Download Patch File Download Diff File Expand all files Collapse all files

66
tests/e2e/churn.js
View File

@@ -32,7 +32,8 @@
// --doc-size <n[k|m]> payload bytes per document (default 16k)
// --index create one secondary index over a churned field
// --mode <m> delete-refill (default) | update
// --rounds <n> delete-refill rounds (default 6)
// --rounds <n> rounds in each mode; update mode splits its writes
// across them (default 6)
// --multiple <n> update mode: total writes as a multiple of --docs
// --target <x> fail unless the steady-state ratio is at or under x
// --port <n> listen port (default 27320)
@@ -100,7 +101,13 @@ const pick = (n) => Math.floor(rnd() * n);
let server = null;
let serverDead = false;
// Bounded: the listeners below run for the life of the process and only the
// last few lines are ever read, so an unbounded string would hold a rope
// proportional to everything the server ever said.
let serverLog = '';
const noteServerLog = (d) => {
serverLog = (serverLog + d).slice(-65536);
};
function cleanup() {
if (server && !serverDead) {
@@ -119,14 +126,14 @@ function startServer() {
server = spawn(BIN, ['--port', String(opt.port), '--db', DBFILE], {
stdio: ['ignore', 'pipe', 'pipe'],
});
server.stdout.on('data', (d) => (serverLog += d));
server.stderr.on('data', (d) => (serverLog += d));
server.stdout.on('data', noteServerLog);
server.stderr.on('data', noteServerLog);
server.on('error', (e) => reject(new Error(`cannot start ${BIN}: ${e.message}`)));
server.on('exit', (code, sig) => {
// A child that dies must fail the start, or the poll below would find a
// *stale* server on the same port and measure the wrong database.
serverDead = true;
if (code !== null && sig === null) serverLog += `\n[child exited rc=${code}]`;
if (code !== null && sig === null) noteServerLog(`\n[child exited rc=${code}]`);
});
const deadline = Date.now() + 15000;
(async () => {
@@ -135,6 +142,10 @@ function startServer() {
reject(new Error(`server child exited during start (port ${opt.port} busy?)\n${serverLog}`));
return;
}
// A fresh client per attempt, deliberately: a MongoClient whose first
// connect fails tears its topology down and every later command on it
// fails the same way, so reusing one turns "not up yet" into "never
// comes up".
const c = new MongoClient(URL, { serverSelectionTimeoutMS: 1000 });
try {
await c.connect();
@@ -177,23 +188,21 @@ const MB = (n) => (n / (1 << 20)).toFixed(1);
// a run against an older build comparable.
let docBytes = 0;
async function stats(client, coll) {
async function stats(client, count) {
const s = await client.db('admin').command({ serverStatus: 1 });
const m = s.multifora || null;
const num = (v) => Number(v);
const count = await coll.countDocuments({});
return {
live: count * docBytes,
count,
file: dataFileBytes(),
m: m && {
live: num(m.liveBytes),
dead: num(m.deadBytes),
reclaimed: num(m.reclaimedBytes),
runs: num(m.slabRuns),
freeReady: num(m.freeReadyPages),
allocTail: num(m.allocTail),
compactions: num(m.compactions),
live: Number(m.liveBytes),
dead: Number(m.deadBytes),
reclaimed: Number(m.reclaimedBytes),
runs: Number(m.slabRuns),
freeReady: Number(m.freeReadyBytes),
allocTail: Number(m.allocTailBytes),
compactions: Number(m.compactions),
},
};
}
@@ -206,10 +215,10 @@ function report(label, s) {
// had to occupy. The file never shrinks, so `ratio` is a high-water mark
// and cannot come down however well reclamation works; `inUse` is the
// number that moves when it does.
const inUse = (s.m.allocTail - s.m.freeReady) * 4096;
const inUse = s.m.allocTail - s.m.freeReady;
line += ` inUse ${(inUse / s.live).toFixed(2)}x` +
` dead ${MB(s.m.dead)}MB reclaimed ${MB(s.m.reclaimed)}MB` +
` allocTail ${MB(s.m.allocTail * 4096)}MB freeReady ${MB(s.m.freeReady * 4096)}MB` +
` allocTail ${MB(s.m.allocTail)}MB freeReady ${MB(s.m.freeReady)}MB` +
` runs ${s.m.runs} compactions ${s.m.compactions}`;
} else {
line += ' (no multifora section: counters n/a)';
@@ -260,7 +269,7 @@ async function main() {
docBytes = require('mongodb').BSON.serialize(makeDoc(0)).length;
await insertRange(coll, 0, opt.docs);
if (opt.index) await coll.createIndex({ k: 1 });
const base = await stats(client, coll);
const base = await stats(client, opt.docs);
report('loaded', base);
const ratios = [];
@@ -283,14 +292,26 @@ async function main() {
live[at] = live[live.length - 1];
live.pop();
}
// One delete spec per id, not one `$in` over thousands of them. The
// server's planner refuses to use an index for an `$in` wider than
// `index.max_combos` (100), so a 5000-element one falls back to a full
// collection scan that re-filters every document against every member --
// quadratic in `--docs`, and it was the whole of this harness's runtime:
// the documented 40k x 16 KiB gate took 62 s and takes 11 s now, and the
// 150k x 200 B line went from ~480 s to 3 s. Reported ratios are
// unchanged, which is the point: this was the instrument's cost, not the
// database's.
const bs = 5000;
for (let i = 0; i < ids.length; i += bs) {
await coll.deleteMany({ _id: { $in: ids.slice(i, i + bs) } });
await coll.bulkWrite(
ids.slice(i, i + bs).map((id) => ({ deleteOne: { filter: { _id: id } } })),
{ ordered: false },
);
}
await insertRange(coll, nextId, nextId + ids.length);
for (let i = 0; i < ids.length; i++) live.push(nextId + i);
nextId += ids.length;
ratios.push(report(`round ${r + 1}`, await stats(client, coll)));
ratios.push(report(`round ${r + 1}`, await stats(client, opt.docs)));
}
} else {
// The same documents rewritten over and over: every rewrite leaves the old
@@ -309,12 +330,13 @@ async function main() {
await coll.bulkWrite(ops, { ordered: false });
done += ops.length;
}
ratios.push(report(`round ${r + 1}`, await stats(client, coll)));
ratios.push(report(`round ${r + 1}`, await stats(client, opt.docs)));
}
}
const final = await stats(client, coll);
const count = final.count;
// The one real count in the run, and the only one the check below needs.
const count = await coll.countDocuments({});
const final = await stats(client, count);
const elapsed = ((Date.now() - t0) / 1000).toFixed(0);
console.log(`churn: ${count} documents live at the end, ${elapsed}s`);