dev/MultiforaDB
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

db/pager: reclaim what churn abandons

Browse Source 
The churn gate (PLAN D6.2 as amended, D7.4) measured a data file growing
linearly and without bound: 50% churn over six rounds reached 7.2x the live
data and was still climbing when the run was stopped. Three separate bugs,
each of which alone was enough to make reclamation impossible.

**The compaction trigger had been dead since commit 14.** `note_compact`
gated on `log.data_bytes`, which was the right question while the log was
the only copy of the data. A checkpoint now truncates the log, and
`truncate_to_header` zeroes that counter -- so the first gate stopped being
reachable and compaction never fired again. Retarget it at the data file,
where the garbage now lives: `Engine.live_bytes`/`dead_bytes`, in bytes
rather than document counts because a rewrite copies bytes. The engine's
live total is the sum over collections by construction, checked in
`write_catalog`, which walks every collection anyway.

**`stable_pages` is a bound, not a membership test.** `page_mut_cow` asked
`p >= stable_pages`, which is right for tail-bumped pages and wrong for
recycled ones -- they come off the free list *below* the mark and are
nonetheless writable, because two-generation retention means no live image
references them. So every write to a recycled node page copied and freed it
again, and both append cursors (the doc slab, the overflow slab) abandoned
each recycled extent after a single record. Nothing was ever really reused.
Replaced with an exact `unpublished` bit set, cleared at each publish: 32 KiB
per GiB, one load against the 4 KiB copy it avoids.

**First fit let one-page requests dismantle the extents.** Copy-on-write
asks for a single page thousands of times per generation while the doc slab
asks for 2048-page extents; first fit carved a page off the front of the
largest run every time, so the free list drained to empty every generation
with the file still growing by the whole write volume. Best fit keeps the
runs whole -- nothing else wants the one-page holes -- and `publish` now
coalesces adjacent runs, without which the list only ever fragments.

Also: a rebuild publishes twice. One publish moves the abandoned extents
from `pending` to `hold`; the space is not reusable until a second, so the
next rebuild grew the file instead of reusing what the last one freed. Safe
for the reason the delay exists -- what the second publish releases is what
the pre-rebuild image referenced, and that image is no longer the fallback.

Measured, sustained-churn steady state, 40k x 16 KiB documents:

  delete half and refill, 6 rounds   4.10x climbing -> 1.65x flat
  random $set over 5x the collection 3.58x         -> 2.47x flat

Above the 1.3x the amended D6.2 hoped for, and structurally so: a rebuild
needs a whole second copy of the live data before the first can be freed.
The gate's purpose was to decide whether doc-level free lists are needed
post-M0, and this is the answer -- yes, for M1.

Five mutations, each verified red: the numeric mark in `page_mut_cow`, first
fit in `take_free`, dropping `coalesce_free_ready`, dropping
`mark_unpublished`, and dropping the rebuild's second checkpoint.
This commit is contained in:
Alexey Shakhmatov
2026-08-03 22:38:57 +03:00
parent b20ae92cbf
commit 5228ed740a
3 changed files with 500 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files

260
src/pager.zig
View File

@@ -209,8 +209,28 @@ pub const Pager = struct {
/// Pages below this belong to the last published image and must never be
/// stored into (PLAN amendment A1). Zero until a checkpoint publishes one,
/// which is why copy-on-write is inert before then.
///
/// A *bound*, not the membership test: see `unpublished`.
stable_pages: u32,
/// Pages handed out since the last publish, and therefore not referenced by
/// any durable image -- free to be written in place however low their page
/// number is.
///
/// `p >= stable_pages` was used for this and is not the same question. It is
/// right for tail-bumped pages and wrong for recycled ones, which come off
/// the free list *below* the mark and are nonetheless writable. The
/// difference is not cosmetic: with the numeric test, every write to a
/// recycled node page copied it and freed it again, and every recycled slab
/// extent was abandoned after a single document -- so nothing was ever really
/// reused and the file grew without bound under churn. The churn gate
/// measured 7.2x live data over six rounds, still climbing linearly.
///
/// One bit per page: 32 KiB per GiB of database, one load on the COW path
/// against the 4 KiB copy it avoids. Cleared wholesale at each publish,
/// which is exactly when every allocated page becomes part of the image.
unpublished: std.DynamicBitSetUnmanaged,
/// Free pages, in three stages. `ready` may be handed out now; `hold` was
/// freed one generation ago; `pending` was freed during this generation.
///
@@ -281,12 +301,16 @@ pub const Pager = struct {
.loaded = .{},
.generation = 0,
.stable_pages = 0,
.unpublished = .{},
.free_ready = .empty,
.free_hold = .empty,
.free_pending = .empty,
.dirty = if (track_dirty) .empty else {},
.protect_ok = false,
};
// The bit set is the one heap allocation `self` owns before `deinit` can
// be reached: a rejected header returns from the middle of this function.
errdefer self.unpublished.deinit(gpa);
if (created) {
try self.grow_to(page_first_data);
@@ -303,6 +327,7 @@ pub const Pager = struct {
}
pub fn deinit(self: *Pager) void {
self.unpublished.deinit(self.gpa);
self.free_ready.deinit(self.gpa);
self.free_hold.deinit(self.gpa);
self.free_pending.deinit(self.gpa);
@@ -324,13 +349,12 @@ pub const Pager = struct {
/// *updated* use `page_mut_cow`; append-only consumers pass a page they
/// allocated themselves.
///
/// There is deliberately no `p >= stable_pages` assert here, and the reason
/// is worth recording because it looks like an obvious check to add. A page
/// recycled off the free list *is* below the mark and *is* legitimately
/// writable -- it was freed two generations ago and no live image references
/// it any more -- so the page number alone cannot distinguish a violation
/// from a reuse, and a per-write set membership test would cost more than it
/// is worth on the hot path.
/// There is deliberately no assert on the page number here. A page recycled
/// off the free list is below the stable mark and is legitimately writable --
/// it was freed two generations ago and no live image references it any more
/// -- so the page number alone cannot distinguish a violation from a reuse.
/// `unpublished` can, but asserting on it here would only restate what
/// `page_mut_cow` has already decided one frame up.
///
/// The invariant is enforced the two ways PLAN amendment A1 describes
/// instead: structurally, because every write to a tree node goes through
@@ -354,15 +378,30 @@ pub const Pager = struct {
/// The victim goes on the free list, which withholds it for two generations,
/// so the image that still references it stays intact and usable.
pub fn page_mut_cow(self: *Pager, slot: *u32) !*align(page_size) [page_size]u8 {
if (slot.* >= self.stable_pages) return self.page_mut(slot.*);
const fresh = try self.alloc_pages(1);
if (self.is_unpublished(slot.*)) return self.page_mut(slot.*);
const copy = try self.alloc_pages(1);
@memcpy(
@as(*[page_size]u8, @ptrCast(self.page_mut(fresh))),
@as(*[page_size]u8, @ptrCast(self.page_mut(copy))),
@as(*const [page_size]u8, @ptrCast(self.page(slot.*))),
);
try self.free_pages(slot.*, 1);
slot.* = fresh;
return self.page_mut(fresh);
slot.* = copy;
return self.page_mut(copy);
}
/// Whether this page was handed out since the last publish, and so may be
/// written in place. See the `unpublished` field.
pub inline fn is_unpublished(self: *const Pager, p: u32) bool {
return p < self.unpublished.bit_length and self.unpublished.isSet(p);
}
/// The same question for the byte-offset consumers: may an append at `off`
/// land in place, or does its page belong to the durable image? Used by the
/// document slab and the overflow slab, which would otherwise have to guess
/// from the offset (and, using `stable_bytes`, guessed wrong for every
/// recycled extent).
pub inline fn is_unpublished_at(self: *const Pager, off: u64) bool {
return self.is_unpublished(@intCast(off >> page_shift));
}
/// The one page a write may legitimately land on below the stable mark: a
@@ -445,19 +484,60 @@ pub const Pager = struct {
/// Take a run from the free list if one fits, else bump the tail. A recycled
/// page was inside a published image once, so its protection has to be
/// lifted before it is handed out again.
/// Best fit, not first fit. First fit lets single-page requests cannibalise
/// the large runs: copy-on-write asks for one page thousands of times per
/// generation, each carving a page off the front of whatever run comes first,
/// and a 2048-page slab extent is shaved to nothing while never being usable
/// as an extent. That is what the churn gate saw -- the free list draining to
/// zero every generation with the file still growing by the full write volume.
/// Smallest-sufficient keeps the big runs whole for the consumers that need
/// them, and there is nothing else that wants the one-page holes.
fn take_free(self: *Pager, n: u32) ?u32 {
// The `pages == n` early exit is the exact-match shortcut only; removing
// it must leave behaviour identical, and turning the loop into a plain
// first-fit `break` is the mutation the extent test names.
var best: ?usize = null;
for (self.free_ready.items, 0..) |e, i| {
if (e.pages < n) continue;
const first = e.first;
if (e.pages == n) {
_ = self.free_ready.swapRemove(i);
} else {
self.free_ready.items[i] = .{ .first = e.first + n, .pages = e.pages - n };
}
self.unprotect(first, n);
return first;
if (best == null or e.pages < self.free_ready.items[best.?].pages) best = i;
if (e.pages == n) break; // cannot do better than exact
}
return null;
const i = best orelse return null;
const e = self.free_ready.items[i];
const first = e.first;
if (e.pages == n) {
_ = self.free_ready.swapRemove(i);
} else {
self.free_ready.items[i] = .{ .first = e.first + n, .pages = e.pages - n };
}
self.unprotect(first, n);
return first;
}
/// Merge runs that touch, so the holes single-page frees leave behind can add
/// up to an extent again. Without it the free list only ever fragments: every
/// generation returns thousands of one-page copy-on-write victims, and an
/// 8 MiB slab extent request never finds a home however much space is free.
/// Once per publish, over a list whose length is the generation's free count.
fn coalesce_free_ready(self: *Pager) void {
const items = self.free_ready.items;
if (items.len < 2) return;
std.mem.sort(Extent, items, {}, struct {
fn less(_: void, a: Extent, b: Extent) bool {
return a.first < b.first;
}
}.less);
var w: usize = 0;
for (items[1..]) |e| {
const prev = &items[w];
if (prev.first + prev.pages == e.first) {
prev.pages += e.pages;
} else {
w += 1;
items[w] = e;
}
}
self.free_ready.items.len = w + 1;
}
pub fn alloc_pages_assume_reserved(self: *Pager, n: u32) u32 {
@@ -474,12 +554,26 @@ pub const Pager = struct {
// Reuse before growing. Without this the free list is decorative and the
// file grows without bound under churn, because copy-on-write abandons
// every page it touches in every generation (PLAN amendment A2).
if (self.take_free(n)) |recycled| return recycled;
if (self.take_free(n)) |recycled| {
self.mark_unpublished(recycled, n);
return recycled;
}
const first = self.alloc_tail;
self.alloc_tail += n;
self.mark_unpublished(first, n);
return first;
}
fn mark_unpublished(self: *Pager, first: u32, n: u32) void {
// `grow_to` sizes the set to the mapping, and `reserve_pages` has already
// grown the mapping past this run, so the range is in bounds.
assert_msg(
@as(usize, first) + n <= self.unpublished.bit_length,
"allocated a page outside the unpublished set",
);
self.unpublished.setRangeValue(.{ .start = first, .end = first + n }, true);
}
/// Bytes currently allocated, i.e. the extent of the live image.
pub fn allocated_bytes(self: *const Pager) u64 {
return @as(u64, self.alloc_tail) << page_shift;
@@ -584,6 +678,10 @@ pub const Pager = struct {
self.file_pages = new_pages;
}
// Before the mapping grows, so a failure here cannot leave pages
// reachable that the set has no bit for.
try self.unpublished.resize(self.gpa, new_pages, false);
const off: usize = @as(usize, self.mapped_pages) << page_shift;
const len: usize = @intCast(new_len - (@as(u64, self.mapped_pages) << page_shift));
_ = try std.posix.mmap(
@@ -749,6 +847,10 @@ pub const Pager = struct {
self.free_hold.clearRetainingCapacity();
try self.free_hold.appendSlice(self.gpa, self.free_pending.items);
self.free_pending.clearRetainingCapacity();
self.coalesce_free_ready();
// Every page handed out so far is now part of the published image, so
// nothing may be written in place any more until it is allocated afresh.
self.unpublished.setRangeValue(.{ .start = 0, .end = self.unpublished.bit_length }, false);
}
// -- free list ----------------------------------------------------------
@@ -1398,6 +1500,120 @@ test "a page already above the mark is written in place, not copied" {
try testing.expectEqual(before, slot);
}
test "a recycled page is written in place rather than copied again" {
// Mutation check: put `slot.* >= self.stable_pages` back in `page_mut_cow`
// in place of `is_unpublished`. Red -- a recycled page has a low page number
// and would be copied and freed all over again, so nothing is ever really
// reused. That is not a hypothetical: it is what the churn gate measured as
// a data file growing linearly and without bound, at 7.2x live data and
// still climbing when the run was stopped.
var threaded: std.Io.Threaded = .init_single_threaded;
defer threaded.deinit();
const io = threaded.io();
var tp = try TmpPager.init(io, 64 << 20);
defer tp.deinit();
const pg = tp.pg();
// Free a page and let it walk all the way to reusable.
const doomed = try pg.alloc_pages(1);
@memset(pg.page_mut(doomed), 0xAA);
try pg.publish(.{ .seq = 1 });
try pg.free_pages(doomed, 1);
try pg.publish(.{ .seq = 2 });
try pg.publish(.{ .seq = 3 });
try testing.expect(pg.free_ready_pages() >= 1);
// Recycling hands it back, below the stable mark.
var slot = try pg.alloc_pages(1);
try testing.expectEqual(doomed, slot);
try testing.expect(slot < pg.stable_pages);
@memset(pg.page_mut(slot), 0xBB);
// A write through the COW path must land in place: the page is not part of
// any published image, whatever its number.
const tail_before = pg.alloc_tail;
const before = slot;
const p = try pg.page_mut_cow(&slot);
@memset(p, 0xCC);
try testing.expectEqual(before, slot);
try testing.expectEqual(tail_before, pg.alloc_tail);
try testing.expectEqual(@as(u8, 0xCC), pg.page(slot)[0]);
}
test "one-page requests do not carve up the runs the extents need" {
// Mutation check: make `take_free` first-fit again (take the first extent
// with `pages >= n`). Red -- the single-page allocations below shave the
// large run down and the extent request has to grow the file.
//
// This is the shape the churn gate hit: copy-on-write asks for one page
// thousands of times per generation while the document slab asks for
// 2048-page extents, so first fit dismantled every run before an extent
// could use it. The free list drained to empty every generation and the file
// still grew by the whole write volume.
var threaded: std.Io.Threaded = .init_single_threaded;
defer threaded.deinit();
const io = threaded.io();
var tp = try TmpPager.init(io, 64 << 20);
defer tp.deinit();
const pg = tp.pg();
// A large run, plus enough single-page holes to serve the small requests.
// The holes are kept apart from each other and from the run by pages that
// are never freed -- otherwise coalescing merges the lot into one run and the
// test stops being about fit at all.
const run = try pg.alloc_pages(64);
_ = try pg.alloc_pages(1); // separator, never freed
var holes: [8]u32 = undefined;
for (&holes) |*h| {
h.* = try pg.alloc_pages(1);
_ = try pg.alloc_pages(1); // separator, never freed
}
try pg.publish(.{ .seq = 1 });
try pg.free_pages(run, 64);
for (holes) |h| try pg.free_pages(h, 1);
try pg.publish(.{ .seq = 2 });
try pg.publish(.{ .seq = 3 });
// Eight one-page allocations must come out of the eight holes.
for (0..holes.len) |_| _ = try pg.alloc_pages(1);
const tail_before = pg.alloc_tail;
// So the run is still whole and the extent request is served from it.
const reused = try pg.alloc_pages(64);
try testing.expectEqual(run, reused);
try testing.expectEqual(tail_before, pg.alloc_tail);
}
test "freed pages that touch merge back into a usable run" {
// Mutation check: drop the `coalesce_free_ready()` call from `publish`. Red
// -- the four one-page frees below stay four separate holes and the run of
// four has to come from the tail. Over a real workload the free list only
// ever fragments, so an 8 MiB extent request never finds a home however much
// free space has accumulated.
var threaded: std.Io.Threaded = .init_single_threaded;
defer threaded.deinit();
const io = threaded.io();
var tp = try TmpPager.init(io, 64 << 20);
defer tp.deinit();
const pg = tp.pg();
var pages: [4]u32 = undefined;
for (&pages) |*x| x.* = try pg.alloc_pages(1);
try pg.publish(.{ .seq = 1 });
// Freed out of order, one page at a time, which is how copy-on-write frees.
try pg.free_pages(pages[2], 1);
try pg.free_pages(pages[0], 1);
try pg.free_pages(pages[3], 1);
try pg.free_pages(pages[1], 1);
try pg.publish(.{ .seq = 2 });
try pg.publish(.{ .seq = 3 });
const tail_before = pg.alloc_tail;
const run = try pg.alloc_pages(4);
try testing.expectEqual(pages[0], run);
try testing.expectEqual(tail_before, pg.alloc_tail);
}
test "freed pages are recycled rather than growing the file" {
// Copy-on-write abandons every page it touches, in every generation, so
// without reuse a write-heavy workload grows the file by