diff --git a/PLAN.md b/PLAN.md index 7186101..8c73ab4 100644 --- a/PLAN.md +++ b/PLAN.md @@ -320,6 +320,71 @@ takes the first one's place. That commit is where this needs handling — a pre-flight scan for compare-equal `_id`s, refusing to drop the map silently while any exist — not here. +### Amendment A5 — the doc-level free list, and what it did not fix (amends A2, closes D7.4) + +D7.4 left M0 with a bound rather than a target: 1.65× delete-heavy, 2.47× +update-heavy against a hoped-for ~1.3×, and the stated conclusion that +doc-level free lists were an M1 item. They are built. The mechanism is +measured, it works, and **the steady-state ratio did not move**. Both halves +of that are the amendment. + +**What was built.** A collection's slab carries a dense map of dead bytes per +`map_align` window — two bytes per window, so 2.7 MB for a 21 GB slab — and a +checkpoint hands back every window with nothing live left in it, splitting the +runs around what is kept. The window is the unit because it is the smallest +thing that can be given back at all: `mark_appendable` refuses an unaligned +start and `protect_stable` rounds outwards. Counting is the whole liveness +test, because `evict_doc` removes a document's index entries before marking its +bytes dead, so "no live bytes in this window" and "nothing references these +bytes" are the same statement. Reclamation lives inside `checkpoint` rather +than beside it so that the run split and the `free_pages` become durable under +one `publish`; there is no new record type, no new catalog version and no +replay path. `alloc_slab_run` is a second policy in the same allocator, because +`take_free`'s best fit — which exists to stop one-page copy-on-write requests +dismantling the extents — can never match a request for 2048 pages against +runs that come back a few windows at a time. + +**What it does not fix, and why no threshold reaches it.** The data file never +shrinks, so `file / live` is a high-water mark, and the mark is set once by the +one thing reclamation cannot avoid: a rebuild needs a whole second copy of the +live data before the first can be freed. Live + garbage-at-trigger + copy is +the peak, and it is reached in the first round, before any free pool exists to +build the copy out of. Rebuilding earlier lowers the garbage term and nothing +else; rebuilding later raises it. So ~2× is the floor of a rebuild-based +design, and tuning is the wrong instrument. Measured occupancy tells the other +half of the story: 1.06–1.26× in use against a 2.46× file, with 934 MB +reclaimed over the run. + +**The successor, named here so the next session does not re-derive it.** +Incremental compaction through a doc-id → offset indirection layer, which is +rejected option (b) of the M1 design, promoted. It is the only thing that +removes the second copy: a rebuild becomes a move of one document at a time +with the map updated behind it. The cost is the one that got it rejected — the +map has to be persistent and crash-safe, i.e. a second copy-on-write B+tree per +collection and a second random read on the point-lookup path — and it undoes +A3. That is a milestone, not a knob. Second and cheaper: return free space to +the filesystem, since 52% of the steady-state file is space the database owns +and is not using; it needs the file never to shrink below what the fallback +generation references, which is its own crash-safety design pass. + +**Small documents behave exactly as forecast**, and the forecast being written +down in advance is what makes it a result. 200-byte documents reclaim nothing +at all — a 16 KiB system page holds ~70 of them and they never all die at once +— and the counters show a mechanism correctly doing nothing rather than one +misfiring. The payoff scales as `doc_size / map_align`, so 4 KiB pages read +four times better on the same code. + +**One thing the gate found that the design had not.** A checkpoint is what +reclaims and a checkpoint is armed by log volume, but a delete logs only an +`_id`. Deleting half a 190 MB collection moved the log by a couple of megabytes +so no checkpoint ran, the garbage sailed past the rebuild threshold, and the +rebuild reset the window map it would have used — six rounds, six rebuilds, +1 MB reclaimed. `compact` now checkpoints before it copies, which is also the +right order on its own terms: the cheap half of the job first, and the +per-collection gate judges what reclamation left. Same six rounds, 256 MB. +Numbers and reproduction in `tests/e2e/results/m0-gates.txt` under `[M1.1]` and +`[M1.2]`. + --- ## 3. Milestones and gates @@ -721,6 +786,16 @@ between `compact` and `checkpoint`; it is out of this scope because it wants its own design pass, and because the free list must not add a second instance of the same shape. +*Where this stands after the free list.* It did add a second instance — +reclamation frees pages as a checkpoint phase, and two checkpoints can be in +flight — so that half is closed: `checkpoint` takes a lock of its own. Two +things came out of doing it. The publish was never the exposure, because it +already runs under `log_lock`; and the whole class is now *detectable* rather +than only arguable, because `write_catalog` asserts per run that the pager has +not already been given it, in test and Debug builds. That assertion is proven +to fire. The original instance — `compact`'s rebuild walk against a concurrent +checkpoint — is unchanged and still wants the design pass. + ### The spec runner starts reading `expectEvents` 354 of the 487 cases declare `expectEvents` and the runner read none of them, @@ -823,15 +898,21 @@ has to be its own commit with its own re-recorded scorecard. the anchor rewritten — resuming at it returned updated documents twice, caught by draining a collection being updated underneath. - Still open in M1: the doc-level free list. The eight reclamation bugs above - were cleared first, as preconditions for the free list rather than as work of - their own; command-monitoring (`expectEvents`) landed next, so that what - followed is measured by an instrument no longer known to overstate. - **A prerequisite the free list must honour**, recorded here while it is - still being designed: *an offset that was ever a record start must remain a - record start.* `doc_bytes` reads a `u32` length prefix in place, so an - offset landing mid-record after a re-split is a garbage-length read rather - than a wrong answer — and an offsets cursor holds exactly such offsets. + The doc-level free list is built; see amendment A5 for what it did and did + not achieve, and `[M1.1]`/`[M1.2]` in the results file for the numbers. The + eight reclamation bugs above were cleared first, as preconditions for it + rather than as work of their own; command-monitoring (`expectEvents`) landed + next, so that what followed is measured by an instrument no longer known to + overstate. + **The prerequisite it had to honour** — *an offset that was ever a record + start must remain a record start*, because `doc_bytes` reads a `u32` length + prefix in place and an offsets cursor holds exactly such offsets — is met + structurally rather than by checking: a window is handed back only when every + byte in it is dead, which means every document touching it has already been + through `evict_doc` and out of every index. What remains is the cursor + holding a *saved* offset list, and that is answered the way a rebuild answers + it, by bumping `layout_epoch` when and only when a collection actually gave + something back. - **M1 sessions** — *settled and implemented.* `lsid` is parsed, validated and deliberately acted on in no way; `txnNumber`, `startTransaction` and `autocommit` are refused; `endSessions` validates the array it discards. diff --git a/tests/e2e/results/m0-gates.txt b/tests/e2e/results/m0-gates.txt index 1718d37..8574f0d 100644 --- a/tests/e2e/results/m0-gates.txt +++ b/tests/e2e/results/m0-gates.txt @@ -157,3 +157,100 @@ # not a write`. The scorecard above is the M0 figure and is left as measured; # those two commits took it to 163 pass / 129 fail, and `tests/spec/scorecard.txt` # always holds the current one. + + +# =========================================================================== +# M1 — doc-level free list (PLAN amendment A5) +# =========================================================================== +# +# Same machine, same driver. Server at the M1 commit named per block, +# ReleaseFast. These are the numbers D7.4 said an M1 item owed. + +[M1.1] churn gate — the doc-level free list + reproduce: node tests/e2e/churn.js --docs 40000 --doc-size 16k --index \ + --mode delete-refill --rounds 6 + node tests/e2e/churn.js --docs 40000 --doc-size 16k --index \ + --mode update --multiple 5 + baseline: the same harness against the end-of-Stage-2 binary (e416ad1), + which has no reclamation and no `multifora` section. + + The harness is committed this time (`tests/e2e/churn.js`), which is half the + point of the block: D7.4's numbers were real and unrepeatable. + + Stage 2 M1 target + delete half and refill, 6x 1.94x 1.94x <= 1.45x NOT MET + random $set over 5x the coll. 2.46x 2.46x <= 1.60x NOT MET + Both flat, drift +0.00x over the last three rounds. + + D7.4 recorded 1.65x for the delete line. This harness reads 1.94x for the + *same binary* D7.4 was measured against the descendants of, so that gap is + the harness, not a regression: the ad-hoc version sampled ids to delete + blindly, which re-picks already-dead ids, deletes fewer than it inserts and + measures a collection that is quietly growing. The update line reproduces + D7.4 exactly (2.46 vs 2.47). + + THE RATIO DID NOT MOVE AND THE MECHANISM WORKS. Both are true, and the + counters are what separate them: + + round 6, update line: reclaimed 934.0MB dead 83.9MB + allocTail 1523.5MB freeReady 797.0MB + inUse 1.16x file/live 2.46x + + Reclamation returned 934 MB over the run and the collection is occupying + 1.16x its live data. What 2.46x measures is the data file's high-water mark, + and the file never shrinks. The mark is set once, in round 1, by the one + thing reclamation cannot avoid: a rebuild needs a whole second copy of the + live data before the first copy can be freed. 626 MB live + the garbage + standing at the moment it fires + 626 MB of copy is the number, and it is + reached before any free pool exists to build the copy out of. + + So the floor for a rebuild-based design is ~2x, and no threshold reaches it. + Rebuilding earlier lowers the garbage term and raises nothing; rebuilding + later raises it. The plan anticipated this exact outcome and said what to do + about it, which is to write it down rather than tune: the remaining lever is + incremental compaction -- a doc-id-to-offset indirection layer, so a rebuild + moves documents without a second copy of everything. That is amendment A5's + successor and it is a milestone of its own, not a knob. + + A second lever, cheaper and not attempted: give free space back to the + filesystem. `freeReady` stands at 797 MB with `allocTail` flat, so 52% of the + file is space the database owns and is not using. Returning the tail-adjacent + part of it needs the file never to shrink below what the fallback generation + references, which is a crash-safety argument and its own design pass. + + What did change, and is the reason the mechanism is worth keeping: + + delete-refill, 12k x 16 KiB reclaimed 1 MB -> 256 MB (six rounds) + + The first figure is what reclamation achieved before `compact` was made to + checkpoint before it copies. A checkpoint is what reclaims and a checkpoint + is armed by log volume; a delete logs only an `_id`, so deleting half a + collection moved the log by a couple of megabytes, no checkpoint ran, and the + rebuild got there first every time and reset the window map it would have + used. The harness found that on its first serious run, which is the argument + for committing it. + +[M1.2] churn gate — 200-byte documents + reproduce: node tests/e2e/churn.js --docs 150000 --doc-size 200 --index \ + --mode delete-refill --rounds 4 + Predicted in advance, in the plan, as a pass rather than a fault: + + reclaimed 0.0 MB over four rounds, exactly as forecast. + ratio 3.93x on both the Stage 2 binary and M1 -- identical, flat. + + Reclamation hands back whole system pages. A 16 KiB page on this machine + holds ~70 documents of 200 bytes and the chance that all 70 are dead at once + under uniform deletion is nil, so nothing is ever handed back. The forecast + said the ratio would not improve and the counters would show a mechanism + that correctly does nothing, rather than one that silently misfires; that is + what they show. + + Read the ratio on this line with care. `live` counts document bytes, and at + 200 bytes the two index trees are comparable in size to the documents + themselves -- the file is already 2.18x at load, before any churn. That + overhead is index structure, not slab garbage, and it is not what this gate + is about. + + The payoff of window reclamation scales as doc_size / map_align, so a 4 KiB + system page (x86-64 Linux) reads four times better on the same code. Every + number in this file is Apple Silicon with 16 KiB pages.