db/pager: an append resumes inside its extent after a checkpoint
`slab_reserve` and `reserve_overflow` abandoned the rest of their extent whenever a checkpoint froze the page the tail pointed into, and took a fresh 8 MiB one. The comment called the waste "bounded by one extent per collection per checkpoint", which is true per checkpoint and says nothing about the sum: nothing reclaims it except a rebuild, and a rebuild only runs when there is garbage. A pure-insert workload produces none. Measured, 40 collections of inserts with incompressible payloads so the log actually reaches the checkpoint threshold: live data file log 29 MB 340 MB 29 MB 38 MB 542 MB 5 MB <- checkpoint 67 MB 681 MB 33 MB 76 MB 1076 MB 9 MB <- checkpoint 115 MB 1357 MB 14 MB <- checkpoint 11.8x the live data and climbing by ~335 MB per checkpoint (40 x 8 MiB), which would exhaust the 64 GB address-space reservation after roughly 6 GB of real data -- and after ~1.2 GB with 200 collections. `DatabaseTooLarge` on a database that is nowhere near too large. The fix is what the plan called for and never got: round the cursor up to the next *system* page and keep the extent. Only the page holding the live tail is in the published image; the rest of the extent holds nothing referenced by the image or by an index, so `Pager.mark_appendable` hands it back for appending (and unprotects it, since it may sit below the stable mark where `protect_image` made it read-only). System pages rather than 4 KiB ones because writeback tears at the granularity the kernel manages: a 4 KiB store dirties a whole 16 KiB page on Apple Silicon, and tearing there would take out the published bytes sharing it. Same 40 collections after: 340 MB -> 352 MB across three checkpoints, the ratio falling monotonically toward the 8 MiB-per-collection floor. 64,000 documents across 8 collections verified byte-for-byte and after a kill -9. The churn gate is unchanged at 1.65x, big.js at 4 GB unchanged (4.32 GB file, reopen 0.5 s, RSS after reopen 130 MB). Two mutations, verified red: dropping the resume branch (a fresh extent per checkpoint), and rounding to `page_size` instead of `map_align` (the resumed append then shares a system page with the published image).
This commit is contained in:
@@ -458,13 +458,18 @@ pub const Index = struct {
|
||||
if (rec_len > inline_limit) overflow_bytes += rec_len;
|
||||
}
|
||||
if (overflow_bytes == 0) return;
|
||||
// Same rule as the document slab: a checkpoint freezes the page the tail
|
||||
// points into, so a frozen tail means starting a fresh extent rather
|
||||
// than writing inside the durable image.
|
||||
// Same reasoning as the document slab: a recycled extent is below the
|
||||
// stable mark and still writable, so ask whether these bytes are in the
|
||||
// published image rather than where they sit.
|
||||
// Same rules as the document slab, for the same reasons. Ask whether the
|
||||
// bytes are in the published image rather than where they sit, because a
|
||||
// recycled extent is below the stable mark and still writable; and when
|
||||
// the tail's page is frozen, skip to the next system page and keep the
|
||||
// extent rather than abandoning what is left of it.
|
||||
if (self.pager.is_unpublished_at(self.ovf_tail) and self.ovf_tail + overflow_bytes <= self.ovf_end) return;
|
||||
const resumed = std.mem.alignForward(u64, self.ovf_tail, pgr.map_align);
|
||||
if (self.ovf_tail != 0 and resumed + overflow_bytes <= self.ovf_end) {
|
||||
self.pager.mark_appendable(resumed, self.ovf_end);
|
||||
self.ovf_tail = resumed;
|
||||
return;
|
||||
}
|
||||
// One extent for the whole batch, or a bespoke one when a single
|
||||
// record is larger than the standard extent (a BSON string reaches
|
||||
// 16 MB).
|
||||
|
||||
Reference in New Issue
Block a user