plan/results: the M1 churn numbers

Amendment A5 and the `[M1.1]`/`[M1.2]` blocks. What they record is a result
with two halves, and the value is in keeping both:

The mechanism works. 934 MB reclaimed over the update run, occupancy at
1.06-1.26x its live data, and the counters that say so are in `serverStatus`
rather than inferred.

The ratio did not move. 1.94x delete-heavy and 2.46x update-heavy, identical
to the end-of-Stage-2 binary measured with the same harness. `file / live` is
a high-water mark because the data file never shrinks, and the mark is set in
the first round by the one thing reclamation cannot avoid: a rebuild needs a
whole second copy of the live data before the first can be freed. So ~2x is
the floor of a rebuild-based design and no threshold reaches it -- rebuilding
earlier lowers the garbage term and nothing else, rebuilding later raises it.
The plan said in advance what to do if this happened, which was to write it
down rather than tune, and to name incremental compaction through a
doc-id-to-offset indirection layer as the successor. Recorded, with its cost:
a second copy-on-write B+tree per collection, a second random read on point
lookup, and it undoes A3.

A second lever is named that the plan had not: 52% of the steady-state file is
space the database owns and is not using, so returning it to the filesystem is
worth more here than reclaiming harder. It needs the file never to shrink
below what the fallback generation references, which is its own crash-safety
pass.

D7.4's 1.65x for the delete line is corrected to 1.94x, and the correction is
the harness rather than a regression -- the same 1.94x comes out of the binary
that predates any of this work. The old ad-hoc version sampled ids to delete
blindly, which re-picks dead ones, so it deleted fewer documents than it
inserted and measured a collection that was quietly growing. The update line
reproduces D7.4 exactly, 2.46 against 2.47.

200-byte documents reclaim nothing, exactly as forecast, and the forecast
being written down beforehand is what makes that a result instead of a
disappointment. 3.93x on both binaries. Also noted, because the number invites
misreading: at that document size the two index trees are comparable to the
documents themselves and the file is already 2.18x before any churn -- index
structure, not slab garbage.
This commit is contained in:
A.Shakhmatov
2026-08-09 18:08:46 +03:00
parent d492726881
commit 1491a47479
2 changed files with 187 additions and 9 deletions

View File

@@ -157,3 +157,100 @@
# not a write`. The scorecard above is the M0 figure and is left as measured;
# those two commits took it to 163 pass / 129 fail, and `tests/spec/scorecard.txt`
# always holds the current one.
# ===========================================================================
# M1 — doc-level free list (PLAN amendment A5)
# ===========================================================================
#
# Same machine, same driver. Server at the M1 commit named per block,
# ReleaseFast. These are the numbers D7.4 said an M1 item owed.
[M1.1] churn gate — the doc-level free list
reproduce: node tests/e2e/churn.js --docs 40000 --doc-size 16k --index \
--mode delete-refill --rounds 6
node tests/e2e/churn.js --docs 40000 --doc-size 16k --index \
--mode update --multiple 5
baseline: the same harness against the end-of-Stage-2 binary (e416ad1),
which has no reclamation and no `multifora` section.
The harness is committed this time (`tests/e2e/churn.js`), which is half the
point of the block: D7.4's numbers were real and unrepeatable.
Stage 2 M1 target
delete half and refill, 6x 1.94x 1.94x <= 1.45x NOT MET
random $set over 5x the coll. 2.46x 2.46x <= 1.60x NOT MET
Both flat, drift +0.00x over the last three rounds.
D7.4 recorded 1.65x for the delete line. This harness reads 1.94x for the
*same binary* D7.4 was measured against the descendants of, so that gap is
the harness, not a regression: the ad-hoc version sampled ids to delete
blindly, which re-picks already-dead ids, deletes fewer than it inserts and
measures a collection that is quietly growing. The update line reproduces
D7.4 exactly (2.46 vs 2.47).
THE RATIO DID NOT MOVE AND THE MECHANISM WORKS. Both are true, and the
counters are what separate them:
round 6, update line: reclaimed 934.0MB dead 83.9MB
allocTail 1523.5MB freeReady 797.0MB
inUse 1.16x file/live 2.46x
Reclamation returned 934 MB over the run and the collection is occupying
1.16x its live data. What 2.46x measures is the data file's high-water mark,
and the file never shrinks. The mark is set once, in round 1, by the one
thing reclamation cannot avoid: a rebuild needs a whole second copy of the
live data before the first copy can be freed. 626 MB live + the garbage
standing at the moment it fires + 626 MB of copy is the number, and it is
reached before any free pool exists to build the copy out of.
So the floor for a rebuild-based design is ~2x, and no threshold reaches it.
Rebuilding earlier lowers the garbage term and raises nothing; rebuilding
later raises it. The plan anticipated this exact outcome and said what to do
about it, which is to write it down rather than tune: the remaining lever is
incremental compaction -- a doc-id-to-offset indirection layer, so a rebuild
moves documents without a second copy of everything. That is amendment A5's
successor and it is a milestone of its own, not a knob.
A second lever, cheaper and not attempted: give free space back to the
filesystem. `freeReady` stands at 797 MB with `allocTail` flat, so 52% of the
file is space the database owns and is not using. Returning the tail-adjacent
part of it needs the file never to shrink below what the fallback generation
references, which is a crash-safety argument and its own design pass.
What did change, and is the reason the mechanism is worth keeping:
delete-refill, 12k x 16 KiB reclaimed 1 MB -> 256 MB (six rounds)
The first figure is what reclamation achieved before `compact` was made to
checkpoint before it copies. A checkpoint is what reclaims and a checkpoint
is armed by log volume; a delete logs only an `_id`, so deleting half a
collection moved the log by a couple of megabytes, no checkpoint ran, and the
rebuild got there first every time and reset the window map it would have
used. The harness found that on its first serious run, which is the argument
for committing it.
[M1.2] churn gate — 200-byte documents
reproduce: node tests/e2e/churn.js --docs 150000 --doc-size 200 --index \
--mode delete-refill --rounds 4
Predicted in advance, in the plan, as a pass rather than a fault:
reclaimed 0.0 MB over four rounds, exactly as forecast.
ratio 3.93x on both the Stage 2 binary and M1 -- identical, flat.
Reclamation hands back whole system pages. A 16 KiB page on this machine
holds ~70 documents of 200 bytes and the chance that all 70 are dead at once
under uniform deletion is nil, so nothing is ever handed back. The forecast
said the ratio would not improve and the counters would show a mechanism
that correctly does nothing, rather than one that silently misfires; that is
what they show.
Read the ratio on this line with care. `live` counts document bytes, and at
200 bytes the two index trees are comparable in size to the documents
themselves -- the file is already 2.18x at load, before any churn. That
overhead is index structure, not slab garbage, and it is not what this gate
is about.
The payoff of window reclamation scales as doc_size / map_align, so a 4 KiB
system page (x86-64 Linux) reads four times better on the same code. Every
number in this file is Apple Silicon with 16 KiB pages.