plan/results: the M1 churn numbers
Amendment A5 and the `[M1.1]`/`[M1.2]` blocks. What they record is a result with two halves, and the value is in keeping both: The mechanism works. 934 MB reclaimed over the update run, occupancy at 1.06-1.26x its live data, and the counters that say so are in `serverStatus` rather than inferred. The ratio did not move. 1.94x delete-heavy and 2.46x update-heavy, identical to the end-of-Stage-2 binary measured with the same harness. `file / live` is a high-water mark because the data file never shrinks, and the mark is set in the first round by the one thing reclamation cannot avoid: a rebuild needs a whole second copy of the live data before the first can be freed. So ~2x is the floor of a rebuild-based design and no threshold reaches it -- rebuilding earlier lowers the garbage term and nothing else, rebuilding later raises it. The plan said in advance what to do if this happened, which was to write it down rather than tune, and to name incremental compaction through a doc-id-to-offset indirection layer as the successor. Recorded, with its cost: a second copy-on-write B+tree per collection, a second random read on point lookup, and it undoes A3. A second lever is named that the plan had not: 52% of the steady-state file is space the database owns and is not using, so returning it to the filesystem is worth more here than reclaiming harder. It needs the file never to shrink below what the fallback generation references, which is its own crash-safety pass. D7.4's 1.65x for the delete line is corrected to 1.94x, and the correction is the harness rather than a regression -- the same 1.94x comes out of the binary that predates any of this work. The old ad-hoc version sampled ids to delete blindly, which re-picks dead ones, so it deleted fewer documents than it inserted and measured a collection that was quietly growing. The update line reproduces D7.4 exactly, 2.46 against 2.47. 200-byte documents reclaim nothing, exactly as forecast, and the forecast being written down beforehand is what makes that a result instead of a disappointment. 3.93x on both binaries. Also noted, because the number invites misreading: at that document size the two index trees are comparable to the documents themselves and the file is already 2.18x before any churn -- index structure, not slab garbage.
This commit is contained in:
@@ -157,3 +157,100 @@
|
||||
# not a write`. The scorecard above is the M0 figure and is left as measured;
|
||||
# those two commits took it to 163 pass / 129 fail, and `tests/spec/scorecard.txt`
|
||||
# always holds the current one.
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# M1 — doc-level free list (PLAN amendment A5)
|
||||
# ===========================================================================
|
||||
#
|
||||
# Same machine, same driver. Server at the M1 commit named per block,
|
||||
# ReleaseFast. These are the numbers D7.4 said an M1 item owed.
|
||||
|
||||
[M1.1] churn gate — the doc-level free list
|
||||
reproduce: node tests/e2e/churn.js --docs 40000 --doc-size 16k --index \
|
||||
--mode delete-refill --rounds 6
|
||||
node tests/e2e/churn.js --docs 40000 --doc-size 16k --index \
|
||||
--mode update --multiple 5
|
||||
baseline: the same harness against the end-of-Stage-2 binary (e416ad1),
|
||||
which has no reclamation and no `multifora` section.
|
||||
|
||||
The harness is committed this time (`tests/e2e/churn.js`), which is half the
|
||||
point of the block: D7.4's numbers were real and unrepeatable.
|
||||
|
||||
Stage 2 M1 target
|
||||
delete half and refill, 6x 1.94x 1.94x <= 1.45x NOT MET
|
||||
random $set over 5x the coll. 2.46x 2.46x <= 1.60x NOT MET
|
||||
Both flat, drift +0.00x over the last three rounds.
|
||||
|
||||
D7.4 recorded 1.65x for the delete line. This harness reads 1.94x for the
|
||||
*same binary* D7.4 was measured against the descendants of, so that gap is
|
||||
the harness, not a regression: the ad-hoc version sampled ids to delete
|
||||
blindly, which re-picks already-dead ids, deletes fewer than it inserts and
|
||||
measures a collection that is quietly growing. The update line reproduces
|
||||
D7.4 exactly (2.46 vs 2.47).
|
||||
|
||||
THE RATIO DID NOT MOVE AND THE MECHANISM WORKS. Both are true, and the
|
||||
counters are what separate them:
|
||||
|
||||
round 6, update line: reclaimed 934.0MB dead 83.9MB
|
||||
allocTail 1523.5MB freeReady 797.0MB
|
||||
inUse 1.16x file/live 2.46x
|
||||
|
||||
Reclamation returned 934 MB over the run and the collection is occupying
|
||||
1.16x its live data. What 2.46x measures is the data file's high-water mark,
|
||||
and the file never shrinks. The mark is set once, in round 1, by the one
|
||||
thing reclamation cannot avoid: a rebuild needs a whole second copy of the
|
||||
live data before the first copy can be freed. 626 MB live + the garbage
|
||||
standing at the moment it fires + 626 MB of copy is the number, and it is
|
||||
reached before any free pool exists to build the copy out of.
|
||||
|
||||
So the floor for a rebuild-based design is ~2x, and no threshold reaches it.
|
||||
Rebuilding earlier lowers the garbage term and raises nothing; rebuilding
|
||||
later raises it. The plan anticipated this exact outcome and said what to do
|
||||
about it, which is to write it down rather than tune: the remaining lever is
|
||||
incremental compaction -- a doc-id-to-offset indirection layer, so a rebuild
|
||||
moves documents without a second copy of everything. That is amendment A5's
|
||||
successor and it is a milestone of its own, not a knob.
|
||||
|
||||
A second lever, cheaper and not attempted: give free space back to the
|
||||
filesystem. `freeReady` stands at 797 MB with `allocTail` flat, so 52% of the
|
||||
file is space the database owns and is not using. Returning the tail-adjacent
|
||||
part of it needs the file never to shrink below what the fallback generation
|
||||
references, which is a crash-safety argument and its own design pass.
|
||||
|
||||
What did change, and is the reason the mechanism is worth keeping:
|
||||
|
||||
delete-refill, 12k x 16 KiB reclaimed 1 MB -> 256 MB (six rounds)
|
||||
|
||||
The first figure is what reclamation achieved before `compact` was made to
|
||||
checkpoint before it copies. A checkpoint is what reclaims and a checkpoint
|
||||
is armed by log volume; a delete logs only an `_id`, so deleting half a
|
||||
collection moved the log by a couple of megabytes, no checkpoint ran, and the
|
||||
rebuild got there first every time and reset the window map it would have
|
||||
used. The harness found that on its first serious run, which is the argument
|
||||
for committing it.
|
||||
|
||||
[M1.2] churn gate — 200-byte documents
|
||||
reproduce: node tests/e2e/churn.js --docs 150000 --doc-size 200 --index \
|
||||
--mode delete-refill --rounds 4
|
||||
Predicted in advance, in the plan, as a pass rather than a fault:
|
||||
|
||||
reclaimed 0.0 MB over four rounds, exactly as forecast.
|
||||
ratio 3.93x on both the Stage 2 binary and M1 -- identical, flat.
|
||||
|
||||
Reclamation hands back whole system pages. A 16 KiB page on this machine
|
||||
holds ~70 documents of 200 bytes and the chance that all 70 are dead at once
|
||||
under uniform deletion is nil, so nothing is ever handed back. The forecast
|
||||
said the ratio would not improve and the counters would show a mechanism
|
||||
that correctly does nothing, rather than one that silently misfires; that is
|
||||
what they show.
|
||||
|
||||
Read the ratio on this line with care. `live` counts document bytes, and at
|
||||
200 bytes the two index trees are comparable in size to the documents
|
||||
themselves -- the file is already 2.18x at load, before any churn. That
|
||||
overhead is index structure, not slab garbage, and it is not what this gate
|
||||
is about.
|
||||
|
||||
The payoff of window reclamation scales as doc_size / map_align, so a 4 KiB
|
||||
system page (x86-64 Linux) reads four times better on the same code. Every
|
||||
number in this file is Apple Silicon with 16 KiB pages.
|
||||
|
||||
Reference in New Issue
Block a user