db/pager: reclaim what churn abandons
The churn gate (PLAN D6.2 as amended, D7.4) measured a data file growing linearly and without bound: 50% churn over six rounds reached 7.2x the live data and was still climbing when the run was stopped. Three separate bugs, each of which alone was enough to make reclamation impossible. **The compaction trigger had been dead since commit 14.** `note_compact` gated on `log.data_bytes`, which was the right question while the log was the only copy of the data. A checkpoint now truncates the log, and `truncate_to_header` zeroes that counter -- so the first gate stopped being reachable and compaction never fired again. Retarget it at the data file, where the garbage now lives: `Engine.live_bytes`/`dead_bytes`, in bytes rather than document counts because a rewrite copies bytes. The engine's live total is the sum over collections by construction, checked in `write_catalog`, which walks every collection anyway. **`stable_pages` is a bound, not a membership test.** `page_mut_cow` asked `p >= stable_pages`, which is right for tail-bumped pages and wrong for recycled ones -- they come off the free list *below* the mark and are nonetheless writable, because two-generation retention means no live image references them. So every write to a recycled node page copied and freed it again, and both append cursors (the doc slab, the overflow slab) abandoned each recycled extent after a single record. Nothing was ever really reused. Replaced with an exact `unpublished` bit set, cleared at each publish: 32 KiB per GiB, one load against the 4 KiB copy it avoids. **First fit let one-page requests dismantle the extents.** Copy-on-write asks for a single page thousands of times per generation while the doc slab asks for 2048-page extents; first fit carved a page off the front of the largest run every time, so the free list drained to empty every generation with the file still growing by the whole write volume. Best fit keeps the runs whole -- nothing else wants the one-page holes -- and `publish` now coalesces adjacent runs, without which the list only ever fragments. Also: a rebuild publishes twice. One publish moves the abandoned extents from `pending` to `hold`; the space is not reusable until a second, so the next rebuild grew the file instead of reusing what the last one freed. Safe for the reason the delay exists -- what the second publish releases is what the pre-rebuild image referenced, and that image is no longer the fallback. Measured, sustained-churn steady state, 40k x 16 KiB documents: delete half and refill, 6 rounds 4.10x climbing -> 1.65x flat random $set over 5x the collection 3.58x -> 2.47x flat Above the 1.3x the amended D6.2 hoped for, and structurally so: a rebuild needs a whole second copy of the live data before the first can be freed. The gate's purpose was to decide whether doc-level free lists are needed post-M0, and this is the answer -- yes, for M1. Five mutations, each verified red: the numeric mark in `page_mut_cow`, first fit in `take_free`, dropping `coalesce_free_ready`, dropping `mark_unpublished`, and dropping the rebuild's second checkpoint.
This commit is contained in:
@@ -456,7 +456,10 @@ pub const Index = struct {
|
||||
// Same rule as the document slab: a checkpoint freezes the page the tail
|
||||
// points into, so a frozen tail means starting a fresh extent rather
|
||||
// than writing inside the durable image.
|
||||
if (self.ovf_tail >= self.pager.stable_bytes() and self.ovf_tail + overflow_bytes <= self.ovf_end) return;
|
||||
// Same reasoning as the document slab: a recycled extent is below the
|
||||
// stable mark and still writable, so ask whether these bytes are in the
|
||||
// published image rather than where they sit.
|
||||
if (self.pager.is_unpublished_at(self.ovf_tail) and self.ovf_tail + overflow_bytes <= self.ovf_end) return;
|
||||
// One extent for the whole batch, or a bespoke one when a single
|
||||
// record is larger than the standard extent (a BSON string reaches
|
||||
// 16 MB).
|
||||
|
||||
Reference in New Issue
Block a user