From 1491a474795d21a5707cbcd57a54c06ffea997f3 Mon Sep 17 00:00:00 2001 From: "A.Shakhmatov" Date: Sun, 9 Aug 2026 18:08:46 +0300 Subject: [PATCH] plan/results: the M1 churn numbers Amendment A5 and the `[M1.1]`/`[M1.2]` blocks. What they record is a result with two halves, and the value is in keeping both: The mechanism works. 934 MB reclaimed over the update run, occupancy at 1.06-1.26x its live data, and the counters that say so are in `serverStatus` rather than inferred. The ratio did not move. 1.94x delete-heavy and 2.46x update-heavy, identical to the end-of-Stage-2 binary measured with the same harness. `file / live` is a high-water mark because the data file never shrinks, and the mark is set in the first round by the one thing reclamation cannot avoid: a rebuild needs a whole second copy of the live data before the first can be freed. So ~2x is the floor of a rebuild-based design and no threshold reaches it -- rebuilding earlier lowers the garbage term and nothing else, rebuilding later raises it. The plan said in advance what to do if this happened, which was to write it down rather than tune, and to name incremental compaction through a doc-id-to-offset indirection layer as the successor. Recorded, with its cost: a second copy-on-write B+tree per collection, a second random read on point lookup, and it undoes A3. A second lever is named that the plan had not: 52% of the steady-state file is space the database owns and is not using, so returning it to the filesystem is worth more here than reclaiming harder. It needs the file never to shrink below what the fallback generation references, which is its own crash-safety pass. D7.4's 1.65x for the delete line is corrected to 1.94x, and the correction is the harness rather than a regression -- the same 1.94x comes out of the binary that predates any of this work. The old ad-hoc version sampled ids to delete blindly, which re-picks dead ones, so it deleted fewer documents than it inserted and measured a collection that was quietly growing. The update line reproduces D7.4 exactly, 2.46 against 2.47. 200-byte documents reclaim nothing, exactly as forecast, and the forecast being written down beforehand is what makes that a result instead of a disappointment. 3.93x on both binaries. Also noted, because the number invites misreading: at that document size the two index trees are comparable to the documents themselves and the file is already 2.18x before any churn -- index structure, not slab garbage. --- PLAN.md | 99 ++++++++++++++++++++++++++++++---- tests/e2e/results/m0-gates.txt | 97 +++++++++++++++++++++++++++++++++ 2 files changed, 187 insertions(+), 9 deletions(-) diff --git a/PLAN.md b/PLAN.md index 7186101..8c73ab4 100644 --- a/PLAN.md +++ b/PLAN.md @@ -320,6 +320,71 @@ takes the first one's place. That commit is where this needs handling — a pre-flight scan for compare-equal `_id`s, refusing to drop the map silently while any exist — not here. +### Amendment A5 — the doc-level free list, and what it did not fix (amends A2, closes D7.4) + +D7.4 left M0 with a bound rather than a target: 1.65× delete-heavy, 2.47× +update-heavy against a hoped-for ~1.3×, and the stated conclusion that +doc-level free lists were an M1 item. They are built. The mechanism is +measured, it works, and **the steady-state ratio did not move**. Both halves +of that are the amendment. + +**What was built.** A collection's slab carries a dense map of dead bytes per +`map_align` window — two bytes per window, so 2.7 MB for a 21 GB slab — and a +checkpoint hands back every window with nothing live left in it, splitting the +runs around what is kept. The window is the unit because it is the smallest +thing that can be given back at all: `mark_appendable` refuses an unaligned +start and `protect_stable` rounds outwards. Counting is the whole liveness +test, because `evict_doc` removes a document's index entries before marking its +bytes dead, so "no live bytes in this window" and "nothing references these +bytes" are the same statement. Reclamation lives inside `checkpoint` rather +than beside it so that the run split and the `free_pages` become durable under +one `publish`; there is no new record type, no new catalog version and no +replay path. `alloc_slab_run` is a second policy in the same allocator, because +`take_free`'s best fit — which exists to stop one-page copy-on-write requests +dismantling the extents — can never match a request for 2048 pages against +runs that come back a few windows at a time. + +**What it does not fix, and why no threshold reaches it.** The data file never +shrinks, so `file / live` is a high-water mark, and the mark is set once by the +one thing reclamation cannot avoid: a rebuild needs a whole second copy of the +live data before the first can be freed. Live + garbage-at-trigger + copy is +the peak, and it is reached in the first round, before any free pool exists to +build the copy out of. Rebuilding earlier lowers the garbage term and nothing +else; rebuilding later raises it. So ~2× is the floor of a rebuild-based +design, and tuning is the wrong instrument. Measured occupancy tells the other +half of the story: 1.06–1.26× in use against a 2.46× file, with 934 MB +reclaimed over the run. + +**The successor, named here so the next session does not re-derive it.** +Incremental compaction through a doc-id → offset indirection layer, which is +rejected option (b) of the M1 design, promoted. It is the only thing that +removes the second copy: a rebuild becomes a move of one document at a time +with the map updated behind it. The cost is the one that got it rejected — the +map has to be persistent and crash-safe, i.e. a second copy-on-write B+tree per +collection and a second random read on the point-lookup path — and it undoes +A3. That is a milestone, not a knob. Second and cheaper: return free space to +the filesystem, since 52% of the steady-state file is space the database owns +and is not using; it needs the file never to shrink below what the fallback +generation references, which is its own crash-safety design pass. + +**Small documents behave exactly as forecast**, and the forecast being written +down in advance is what makes it a result. 200-byte documents reclaim nothing +at all — a 16 KiB system page holds ~70 of them and they never all die at once +— and the counters show a mechanism correctly doing nothing rather than one +misfiring. The payoff scales as `doc_size / map_align`, so 4 KiB pages read +four times better on the same code. + +**One thing the gate found that the design had not.** A checkpoint is what +reclaims and a checkpoint is armed by log volume, but a delete logs only an +`_id`. Deleting half a 190 MB collection moved the log by a couple of megabytes +so no checkpoint ran, the garbage sailed past the rebuild threshold, and the +rebuild reset the window map it would have used — six rounds, six rebuilds, +1 MB reclaimed. `compact` now checkpoints before it copies, which is also the +right order on its own terms: the cheap half of the job first, and the +per-collection gate judges what reclamation left. Same six rounds, 256 MB. +Numbers and reproduction in `tests/e2e/results/m0-gates.txt` under `[M1.1]` and +`[M1.2]`. + --- ## 3. Milestones and gates @@ -721,6 +786,16 @@ between `compact` and `checkpoint`; it is out of this scope because it wants its own design pass, and because the free list must not add a second instance of the same shape. +*Where this stands after the free list.* It did add a second instance — +reclamation frees pages as a checkpoint phase, and two checkpoints can be in +flight — so that half is closed: `checkpoint` takes a lock of its own. Two +things came out of doing it. The publish was never the exposure, because it +already runs under `log_lock`; and the whole class is now *detectable* rather +than only arguable, because `write_catalog` asserts per run that the pager has +not already been given it, in test and Debug builds. That assertion is proven +to fire. The original instance — `compact`'s rebuild walk against a concurrent +checkpoint — is unchanged and still wants the design pass. + ### The spec runner starts reading `expectEvents` 354 of the 487 cases declare `expectEvents` and the runner read none of them, @@ -823,15 +898,21 @@ has to be its own commit with its own re-recorded scorecard. the anchor rewritten — resuming at it returned updated documents twice, caught by draining a collection being updated underneath. - Still open in M1: the doc-level free list. The eight reclamation bugs above - were cleared first, as preconditions for the free list rather than as work of - their own; command-monitoring (`expectEvents`) landed next, so that what - followed is measured by an instrument no longer known to overstate. - **A prerequisite the free list must honour**, recorded here while it is - still being designed: *an offset that was ever a record start must remain a - record start.* `doc_bytes` reads a `u32` length prefix in place, so an - offset landing mid-record after a re-split is a garbage-length read rather - than a wrong answer — and an offsets cursor holds exactly such offsets. + The doc-level free list is built; see amendment A5 for what it did and did + not achieve, and `[M1.1]`/`[M1.2]` in the results file for the numbers. The + eight reclamation bugs above were cleared first, as preconditions for it + rather than as work of their own; command-monitoring (`expectEvents`) landed + next, so that what followed is measured by an instrument no longer known to + overstate. + **The prerequisite it had to honour** — *an offset that was ever a record + start must remain a record start*, because `doc_bytes` reads a `u32` length + prefix in place and an offsets cursor holds exactly such offsets — is met + structurally rather than by checking: a window is handed back only when every + byte in it is dead, which means every document touching it has already been + through `evict_doc` and out of every index. What remains is the cursor + holding a *saved* offset list, and that is answered the way a rebuild answers + it, by bumping `layout_epoch` when and only when a collection actually gave + something back. - **M1 sessions** — *settled and implemented.* `lsid` is parsed, validated and deliberately acted on in no way; `txnNumber`, `startTransaction` and `autocommit` are refused; `endSessions` validates the array it discards. diff --git a/tests/e2e/results/m0-gates.txt b/tests/e2e/results/m0-gates.txt index 1718d37..8574f0d 100644 --- a/tests/e2e/results/m0-gates.txt +++ b/tests/e2e/results/m0-gates.txt @@ -157,3 +157,100 @@ # not a write`. The scorecard above is the M0 figure and is left as measured; # those two commits took it to 163 pass / 129 fail, and `tests/spec/scorecard.txt` # always holds the current one. + + +# =========================================================================== +# M1 — doc-level free list (PLAN amendment A5) +# =========================================================================== +# +# Same machine, same driver. Server at the M1 commit named per block, +# ReleaseFast. These are the numbers D7.4 said an M1 item owed. + +[M1.1] churn gate — the doc-level free list + reproduce: node tests/e2e/churn.js --docs 40000 --doc-size 16k --index \ + --mode delete-refill --rounds 6 + node tests/e2e/churn.js --docs 40000 --doc-size 16k --index \ + --mode update --multiple 5 + baseline: the same harness against the end-of-Stage-2 binary (e416ad1), + which has no reclamation and no `multifora` section. + + The harness is committed this time (`tests/e2e/churn.js`), which is half the + point of the block: D7.4's numbers were real and unrepeatable. + + Stage 2 M1 target + delete half and refill, 6x 1.94x 1.94x <= 1.45x NOT MET + random $set over 5x the coll. 2.46x 2.46x <= 1.60x NOT MET + Both flat, drift +0.00x over the last three rounds. + + D7.4 recorded 1.65x for the delete line. This harness reads 1.94x for the + *same binary* D7.4 was measured against the descendants of, so that gap is + the harness, not a regression: the ad-hoc version sampled ids to delete + blindly, which re-picks already-dead ids, deletes fewer than it inserts and + measures a collection that is quietly growing. The update line reproduces + D7.4 exactly (2.46 vs 2.47). + + THE RATIO DID NOT MOVE AND THE MECHANISM WORKS. Both are true, and the + counters are what separate them: + + round 6, update line: reclaimed 934.0MB dead 83.9MB + allocTail 1523.5MB freeReady 797.0MB + inUse 1.16x file/live 2.46x + + Reclamation returned 934 MB over the run and the collection is occupying + 1.16x its live data. What 2.46x measures is the data file's high-water mark, + and the file never shrinks. The mark is set once, in round 1, by the one + thing reclamation cannot avoid: a rebuild needs a whole second copy of the + live data before the first copy can be freed. 626 MB live + the garbage + standing at the moment it fires + 626 MB of copy is the number, and it is + reached before any free pool exists to build the copy out of. + + So the floor for a rebuild-based design is ~2x, and no threshold reaches it. + Rebuilding earlier lowers the garbage term and raises nothing; rebuilding + later raises it. The plan anticipated this exact outcome and said what to do + about it, which is to write it down rather than tune: the remaining lever is + incremental compaction -- a doc-id-to-offset indirection layer, so a rebuild + moves documents without a second copy of everything. That is amendment A5's + successor and it is a milestone of its own, not a knob. + + A second lever, cheaper and not attempted: give free space back to the + filesystem. `freeReady` stands at 797 MB with `allocTail` flat, so 52% of the + file is space the database owns and is not using. Returning the tail-adjacent + part of it needs the file never to shrink below what the fallback generation + references, which is a crash-safety argument and its own design pass. + + What did change, and is the reason the mechanism is worth keeping: + + delete-refill, 12k x 16 KiB reclaimed 1 MB -> 256 MB (six rounds) + + The first figure is what reclamation achieved before `compact` was made to + checkpoint before it copies. A checkpoint is what reclaims and a checkpoint + is armed by log volume; a delete logs only an `_id`, so deleting half a + collection moved the log by a couple of megabytes, no checkpoint ran, and the + rebuild got there first every time and reset the window map it would have + used. The harness found that on its first serious run, which is the argument + for committing it. + +[M1.2] churn gate — 200-byte documents + reproduce: node tests/e2e/churn.js --docs 150000 --doc-size 200 --index \ + --mode delete-refill --rounds 4 + Predicted in advance, in the plan, as a pass rather than a fault: + + reclaimed 0.0 MB over four rounds, exactly as forecast. + ratio 3.93x on both the Stage 2 binary and M1 -- identical, flat. + + Reclamation hands back whole system pages. A 16 KiB page on this machine + holds ~70 documents of 200 bytes and the chance that all 70 are dead at once + under uniform deletion is nil, so nothing is ever handed back. The forecast + said the ratio would not improve and the counters would show a mechanism + that correctly does nothing, rather than one that silently misfires; that is + what they show. + + Read the ratio on this line with care. `live` counts document bytes, and at + 200 bytes the two index trees are comparable in size to the documents + themselves -- the file is already 2.18x at load, before any churn. That + overhead is index structure, not slab garbage, and it is not what this gate + is about. + + The payoff of window reclamation scales as doc_size / map_align, so a 4 KiB + system page (x86-64 Linux) reads four times better on the same code. Every + number in this file is Apple Silicon with 16 KiB pages.