db/pager: an append resumes inside its extent after a checkpoint

`slab_reserve` and `reserve_overflow` abandoned the rest of their extent
whenever a checkpoint froze the page the tail pointed into, and took a fresh
8 MiB one. The comment called the waste "bounded by one extent per collection
per checkpoint", which is true per checkpoint and says nothing about the sum:
nothing reclaims it except a rebuild, and a rebuild only runs when there is
garbage. A pure-insert workload produces none.

Measured, 40 collections of inserts with incompressible payloads so the log
actually reaches the checkpoint threshold:

  live    data file   log
   29 MB     340 MB   29 MB
   38 MB     542 MB    5 MB   <- checkpoint
   67 MB     681 MB   33 MB
   76 MB    1076 MB    9 MB   <- checkpoint
  115 MB    1357 MB   14 MB   <- checkpoint

11.8x the live data and climbing by ~335 MB per checkpoint (40 x 8 MiB), which
would exhaust the 64 GB address-space reservation after roughly 6 GB of real
data -- and after ~1.2 GB with 200 collections. `DatabaseTooLarge` on a
database that is nowhere near too large.

The fix is what the plan called for and never got: round the cursor up to the
next *system* page and keep the extent. Only the page holding the live tail is
in the published image; the rest of the extent holds nothing referenced by the
image or by an index, so `Pager.mark_appendable` hands it back for appending
(and unprotects it, since it may sit below the stable mark where
`protect_image` made it read-only). System pages rather than 4 KiB ones because
writeback tears at the granularity the kernel manages: a 4 KiB store dirties a
whole 16 KiB page on Apple Silicon, and tearing there would take out the
published bytes sharing it.

Same 40 collections after: 340 MB -> 352 MB across three checkpoints, the ratio
falling monotonically toward the 8 MiB-per-collection floor. 64,000 documents
across 8 collections verified byte-for-byte and after a kill -9. The churn gate
is unchanged at 1.65x, big.js at 4 GB unchanged (4.32 GB file, reopen 0.5 s,
RSS after reopen 130 MB).

Two mutations, verified red: dropping the resume branch (a fresh extent per
checkpoint), and rounding to `page_size` instead of `map_align` (the resumed
append then shares a system page with the published image).
This commit is contained in:
2026-08-04 00:39:44 +03:00
parent 6c7f1f2e77
commit 1814020df9
3 changed files with 146 additions and 6 deletions

View File

@@ -458,13 +458,18 @@ pub const Index = struct {
if (rec_len > inline_limit) overflow_bytes += rec_len;
}
if (overflow_bytes == 0) return;
// Same rule as the document slab: a checkpoint freezes the page the tail
// points into, so a frozen tail means starting a fresh extent rather
// than writing inside the durable image.
// Same reasoning as the document slab: a recycled extent is below the
// stable mark and still writable, so ask whether these bytes are in the
// published image rather than where they sit.
// Same rules as the document slab, for the same reasons. Ask whether the
// bytes are in the published image rather than where they sit, because a
// recycled extent is below the stable mark and still writable; and when
// the tail's page is frozen, skip to the next system page and keep the
// extent rather than abandoning what is left of it.
if (self.pager.is_unpublished_at(self.ovf_tail) and self.ovf_tail + overflow_bytes <= self.ovf_end) return;
const resumed = std.mem.alignForward(u64, self.ovf_tail, pgr.map_align);
if (self.ovf_tail != 0 and resumed + overflow_bytes <= self.ovf_end) {
self.pager.mark_appendable(resumed, self.ovf_end);
self.ovf_tail = resumed;
return;
}
// One extent for the whole batch, or a bespoke one when a single
// record is larger than the standard extent (a BSON string reaches
// 16 MB).