dev/MultiforaDB
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

db/pager: reclaim what churn abandons

Browse Source 
The churn gate (PLAN D6.2 as amended, D7.4) measured a data file growing
linearly and without bound: 50% churn over six rounds reached 7.2x the live
data and was still climbing when the run was stopped. Three separate bugs,
each of which alone was enough to make reclamation impossible.

**The compaction trigger had been dead since commit 14.** `note_compact`
gated on `log.data_bytes`, which was the right question while the log was
the only copy of the data. A checkpoint now truncates the log, and
`truncate_to_header` zeroes that counter -- so the first gate stopped being
reachable and compaction never fired again. Retarget it at the data file,
where the garbage now lives: `Engine.live_bytes`/`dead_bytes`, in bytes
rather than document counts because a rewrite copies bytes. The engine's
live total is the sum over collections by construction, checked in
`write_catalog`, which walks every collection anyway.

**`stable_pages` is a bound, not a membership test.** `page_mut_cow` asked
`p >= stable_pages`, which is right for tail-bumped pages and wrong for
recycled ones -- they come off the free list *below* the mark and are
nonetheless writable, because two-generation retention means no live image
references them. So every write to a recycled node page copied and freed it
again, and both append cursors (the doc slab, the overflow slab) abandoned
each recycled extent after a single record. Nothing was ever really reused.
Replaced with an exact `unpublished` bit set, cleared at each publish: 32 KiB
per GiB, one load against the 4 KiB copy it avoids.

**First fit let one-page requests dismantle the extents.** Copy-on-write
asks for a single page thousands of times per generation while the doc slab
asks for 2048-page extents; first fit carved a page off the front of the
largest run every time, so the free list drained to empty every generation
with the file still growing by the whole write volume. Best fit keeps the
runs whole -- nothing else wants the one-page holes -- and `publish` now
coalesces adjacent runs, without which the list only ever fragments.

Also: a rebuild publishes twice. One publish moves the abandoned extents
from `pending` to `hold`; the space is not reusable until a second, so the
next rebuild grew the file instead of reusing what the last one freed. Safe
for the reason the delay exists -- what the second publish releases is what
the pre-rebuild image referenced, and that image is no longer the fallback.

Measured, sustained-churn steady state, 40k x 16 KiB documents:

  delete half and refill, 6 rounds   4.10x climbing -> 1.65x flat
  random $set over 5x the collection 3.58x         -> 2.47x flat

Above the 1.3x the amended D6.2 hoped for, and structurally so: a rebuild
needs a whole second copy of the live data before the first can be freed.
The gate's purpose was to decide whether doc-level free lists are needed
post-M0, and this is the answer -- yes, for M1.

Five mutations, each verified red: the numeric mark in `page_mut_cow`, first
fit in `take_free`, dropping `coalesce_free_ready`, dropping
`mark_unpublished`, and dropping the rebuild's second checkpoint.
This commit is contained in:
Alexey Shakhmatov
2026-08-03 22:38:57 +03:00
parent b20ae92cbf
commit 5228ed740a
3 changed files with 500 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files

275
src/db.zig
View File

@@ -50,6 +50,11 @@ pub const Collection = struct {
/// rebuild. `slab_tail` cannot answer that -- it is an absolute file offset,
/// so it jumps forward whenever a fresh extent is taken.
slab_used: u64,
/// Of those bytes, the ones still reachable. `slab_used - live_bytes` is
/// this collection's slab garbage, which only a rebuild reclaims. Kept per
/// collection so dropping one can move the right amount from the engine's
/// live total to its dead total.
live_bytes: u64,
/// Secondary indexes (persisted through the log). Heap-allocated, so an
/// `*Index` handed out by `find_index` or `create_index` stays valid when
/// a sibling index is dropped. Held by value, `orderedRemove` memmoved the
@@ -85,6 +90,7 @@ pub const Collection = struct {
.slab_tail = 0,
.slab_end = 0,
.slab_used = 0,
.live_bytes = 0,
.indexes = .empty,
.id_index = undefined,
};
@@ -122,7 +128,12 @@ pub const Collection = struct {
// page the tail points into. Appending there would store inside the
// durable image, so abandon the rest of the extent and start a fresh
// one. The waste is bounded by one extent per collection per checkpoint.
if (self.slab_tail >= self.pager.stable_bytes() and self.slab_tail + len <= self.slab_end) return;
//
// `is_unpublished_at` rather than a comparison against the stable mark:
// an extent recycled off the free list starts *below* the mark and is
// still writable. Asking the mark meant every recycled extent was thrown
// away after one document, so churn never reused anything.
if (self.pager.is_unpublished_at(self.slab_tail) and self.slab_tail + len <= self.slab_end) return;
// A document larger than the standard extent gets one of its own; BSON
// reaches 16 MB and the extent is 8 MiB.
const want_pages: u32 = @intCast(@max(
@@ -147,6 +158,9 @@ pub const Collection = struct {
@memcpy(self.pager.bytes_mut(off, bytes.len), bytes);
self.slab_tail += bytes.len;
self.slab_used += bytes.len;
// Here rather than at the call site: a rebuild appends through this same
// path, and its copies are live by definition.
self.live_bytes += bytes.len;
return off;
}
@@ -240,6 +254,13 @@ pub const Engine = struct {
/// rewrite is worth doing — see `note_compact`.
live_docs: u64 = 0,
dead_docs: u64 = 0,
/// The same question in bytes, about the *data file* rather than the log.
/// Once a checkpoint truncates the log, the log no longer holds the garbage
/// -- the doc slab does, and only a rebuild reclaims it. These are what
/// `note_compact` gates on; counting documents would let one collection of
/// 16 KiB documents and one of 40 B documents look identical.
live_bytes: u64 = 0,
dead_bytes: u64 = 0,
/// Set when the log has grown enough since the last checkpoint to be worth
/// reclaiming. Read by the write epilogue and the TTL monitor, both of which
/// run without holding a collection lock.
@@ -314,6 +335,9 @@ pub const Engine = struct {
engine.seq = replay_from;
engine.committed_seq = replay_from;
engine.live_docs = engine.pager.loaded.live_docs;
// Without this a restart forgets its garbage, and a churned
// database would never compact again.
engine.dead_bytes = engine.pager.loaded.dead_bytes;
}
}
@@ -352,6 +376,9 @@ pub const Engine = struct {
assert_msg(self.live_docs >= coll.doc_count, "dropping a collection would underflow the engine's live count");
self.live_docs -= coll.doc_count;
self.dead_docs += coll.doc_count;
assert_msg(self.live_bytes >= coll.live_bytes, "dropping a collection would underflow the engine's live bytes");
self.live_bytes -= coll.live_bytes;
self.dead_bytes += coll.live_bytes;
coll.id_index.deinit(self.gpa);
for (coll.indexes.items) |ix| {
ix.deinit(self.gpa);
@@ -388,6 +415,17 @@ pub const Engine = struct {
/// regenerating them from it, which is far cheaper than scanning.
/// `id_enc` is `bson.encode_key` of the document's `_id` -- the canonical
/// encoding, which is what the `_id_` index is keyed on.
/// Copy a new document's bytes into the collection's slab and count them as
/// live at both levels. `Collection.slab_append` maintains the collection's
/// own total (a rebuild appends through it too, and its copies are live by
/// definition); the engine's total only moves when a document actually
/// becomes live, which a rebuild's copies do not.
fn publish_doc_bytes(self: *Engine, coll: *Collection, bytes: []const u8) u64 {
const off = coll.slab_append(bytes);
self.live_bytes += bytes.len;
return off;
}
fn evict_doc(self: *Engine, coll: *Collection, id_enc: []const u8) void {
const off = coll.id_index.lookup_exact(id_enc) orelse return;
// Resolve the bytes before any mutation; the slab is untouched by
@@ -399,8 +437,13 @@ pub const Engine = struct {
assert_msg(self.live_docs >= 1, "evicting a document would underflow the engine's live count");
self.live_docs -= 1;
self.dead_docs += 1;
assert_msg(self.live_bytes >= old_bytes.len, "evicting a document would underflow the engine's live bytes");
self.live_bytes -= old_bytes.len;
self.dead_bytes += old_bytes.len;
assert_msg(coll.doc_count >= 1, "evicting a document would underflow the collection's count");
coll.doc_count -= 1;
assert_msg(coll.live_bytes >= old_bytes.len, "evicting a document would underflow the collection's live bytes");
coll.live_bytes -= old_bytes.len;
}
/// `bson.encode_key` of a stored document's `_id`, owned by the caller.
@@ -743,7 +786,7 @@ pub const Engine = struct {
// 7. Publish the document and its entries: copy the bytes into the
// slab and record the offset. Infallible from here.
const off = coll.slab_append(doc_bytes);
const off = self.publish_doc_bytes(coll, doc_bytes);
self.live_docs += 1;
coll.doc_count += 1;
for (built_list.items) |*b| {
@@ -1100,21 +1143,24 @@ pub const Engine = struct {
}
fn note_compact(self: *Engine) void {
// The threshold counts data volume (uncompressed record bytes), not
// the on-disk size: a compressed log would otherwise stay under any
// byte threshold and never compact its garbage.
if (self.log.data_bytes < self.compact_threshold) return;
// Only rewrite when enough of the log is actually garbage. The old
// rule fired on bytes appended, which is the wrong question twice
// over: a 1 GB bulk load has no garbage at all yet would compact
// ~64 times under a 16 MiB threshold (rewriting 1 GB each time,
// hence quadratic), while a small collection rewritten in place
// accumulates garbage indefinitely without ever hitting the count.
// Garbage is measured in the *data file*, not the log. This used to read
// `log.data_bytes`, which was the right question while the log was the
// only copy of the data -- but a checkpoint now truncates the log and
// `truncate_to_header` resets that counter, so the first gate stopped
// being reachable and compaction silently never fired again. The churn
// gate caught it: 50% churn over three rounds left the data file at 4.1x
// the live data, with a trigger that had been dead since the log started
// being reclaimed.
//
// Garbage share is dead / (live + dead); this fires at ~20%, so the
// file stays near 1.25x the live data and each compaction is paid
// for by the space it reclaims.
if (self.dead_docs * 4 < self.live_docs) return;
// Absolute volume first: a rewrite costs a full copy of the live data,
// so it is not worth doing for a few kilobytes however bad the ratio.
if (self.dead_bytes < self.compact_threshold) return;
// Then the share, dead / (live + dead), firing at ~20%: the file stays
// near 1.25x the live data and each rebuild is paid for by the space it
// reclaims. Bytes rather than document counts, because a rewrite copies
// bytes -- 100k evicted 40 B documents are not worth the same rebuild as
// 100k evicted 16 KiB ones.
if (self.dead_bytes * 4 < self.live_bytes) return;
self.compact_pending.store(true, .release);
}
@@ -1184,10 +1230,24 @@ pub const Engine = struct {
if (rebuild_err) |err| return err;
self.dead_docs = 0;
// Every collection's slab was just repacked to hold only live bytes.
self.dead_bytes = 0;
// Publish the rebuilt layout, which is also what reclaims the log. Until
// this lands the old pages are still referenced by the previous
// watermark, so a crash mid-rebuild simply loses the rebuild.
try self.checkpoint();
// And again, to walk the pages the rebuild just abandoned the rest of the
// way down the free list: one publish moves them from `pending` to
// `hold`, a second from `hold` to `ready`. Without this the space a
// rebuild reclaims is not reusable until two unrelated checkpoints have
// happened, so the next rebuild grows the file instead of reusing it --
// measured as ~1.2x of extra steady-state size under sustained churn.
//
// Safe for the same reason the delay exists: what the second publish
// releases is the pages the *pre-rebuild* image referenced, and that
// image is no longer the fallback -- the first publish made the rebuilt
// one current and the one before it the fallback. Both remain intact.
try self.checkpoint();
}
/// Copy one collection's live documents into fresh extents and rebuild every
@@ -1209,6 +1269,7 @@ pub const Engine = struct {
coll.slab_tail = 0;
coll.slab_end = 0;
coll.slab_used = 0;
coll.live_bytes = 0;
// Walk in _id order, which is also the order the new slab ends up in --
// so a later scan reads it sequentially.
@@ -1347,6 +1408,13 @@ pub const Engine = struct {
fn write_catalog(self: *Engine, out: *std.ArrayListUnmanaged(u8)) !void {
const gpa = self.gpa;
// The engine's live-byte total is by definition the sum over
// collections, and `read_catalog` rebuilds it that way. Check it here,
// where every collection is being walked regardless: a divergence means
// some path published or evicted bytes at one level and not the other,
// and the visible symptom would be a compaction trigger that fires
// never or always.
var live_sum: u64 = 0;
try put_u32(gpa, out, catalog_magic);
try put_u32(gpa, out, catalog_version);
try put_u64(gpa, out, self.live_docs);
@@ -1363,6 +1431,12 @@ pub const Engine = struct {
try put_u64(gpa, out, coll.slab_tail);
try put_u64(gpa, out, coll.slab_end);
try put_u64(gpa, out, coll.slab_used);
try put_u64(gpa, out, coll.live_bytes);
live_sum += coll.live_bytes;
assert_msg(
coll.live_bytes <= coll.slab_used,
"a collection cannot hold more live bytes than it ever appended",
);
try put_u32(gpa, out, @intCast(coll.slab_extents.items.len));
for (coll.slab_extents.items) |e| {
try put_u32(gpa, out, e.first);
@@ -1373,6 +1447,10 @@ pub const Engine = struct {
for (coll.indexes.items) |ix| try write_index_catalog(gpa, out, ix);
}
}
assert_msg(
live_sum == self.live_bytes,
"the engine's live-byte total must equal the sum over collections",
);
try put_u64(gpa, out, std.hash.XxHash3.hash(0, out.items));
}
@@ -1437,6 +1515,10 @@ pub const Engine = struct {
coll.slab_tail = try r.read_u64();
coll.slab_end = try r.read_u64();
coll.slab_used = try r.read_u64();
coll.live_bytes = try r.read_u64();
// The engine's total is the sum over collections rather than a
// separately stored field, so the two cannot disagree.
self.live_bytes += coll.live_bytes;
const nex = try r.read_u32();
var e: u32 = 0;
while (e < nex) : (e += 1) {
@@ -1570,6 +1652,8 @@ pub const Engine = struct {
self.dbs.clearRetainingCapacity();
self.live_docs = 0;
self.dead_docs = 0;
self.live_bytes = 0;
self.dead_bytes = 0;
self.seq = 0;
self.committed_seq = 0;
}
@@ -1621,6 +1705,7 @@ pub const Engine = struct {
.catalog_page = first,
.catalog_len = buf.items.len,
.live_docs = self.live_docs,
.dead_bytes = self.dead_bytes,
}) catch |err| {
self.log_lock.unlock(self.io);
return err;
@@ -1747,7 +1832,7 @@ fn apply_record(ctx: *anyopaque, record: storage.Record, doc: *bson.Document) an
const doc_bytes = try serialize_doc(self.gpa, doc);
defer self.gpa.free(doc_bytes);
try coll.slab_reserve(self.gpa, doc_bytes.len);
const off = coll.slab_append(doc_bytes);
const off = self.publish_doc_bytes(coll, doc_bytes);
self.live_docs += 1;
coll.doc_count += 1;
// The `_id_` entry is added *now*, not after replay: it is the only
@@ -1939,6 +2024,162 @@ test "compaction reclaims garbage but leaves a garbage-free log alone" {
try testing.expect(engine.log.data_bytes < after_insert * 2);
}
test "compaction still triggers after a checkpoint has truncated the log" {
// Mutation: gate `note_compact` on `self.log.data_bytes` (what it read
// before the checkpoint existed) instead of `self.dead_bytes`. Red, because
// `truncate_to_header` zeroes that counter at every checkpoint -- the trigger
// then never fires and the doc slab grows without bound. This test exists
// because the churn gate measured exactly that: 4.1x live data.
//
// Second mutation: drop `dead_bytes` from the watermark, or from the restore
// beside `live_docs` in `open`. Red on the reopened engine below.
var threaded: std.Io.Threaded = .init_single_threaded;
defer threaded.deinit();
var env = test_env(&threaded);
const io = env.io;
const gpa = testing.allocator;
var tmp = try TmpLog.init(gpa);
defer tmp.deinit(gpa);
var engine = try Engine.open(gpa, io, tmp.path);
defer engine.deinit();
// 200 documents of a few dozen bytes each, so the volume gate has to be
// small enough for their garbage to clear it.
engine.compact_threshold = 1024;
try engine.lock();
defer engine.unlock();
for (0..200) |i| {
var d = try make_doc(gpa, @intCast(i), "x");
defer d.deinit();
try engine.insert("app", "c", &d, &env.gen);
}
try engine.commit();
// A checkpoint, which is what truncates the log. From here on the log says
// nothing about how much garbage the database holds.
try engine.checkpoint();
try testing.expectEqual(@as(u64, 0), engine.log.data_bytes);
const live_after_load = engine.live_bytes;
try testing.expect(live_after_load > 0);
// Now make garbage. Every replace supersedes a document, so its slab bytes
// are dead: the data file holds them and only a rebuild reclaims them.
_ = engine.take_compact();
for (0..200) |i| {
var d = try make_doc(gpa, @intCast(i), "yy");
defer d.deinit();
try engine.replace("app", "c", &d, &env.gen);
}
try engine.commit();
try testing.expectEqual(live_after_load + 200, engine.live_bytes);
try testing.expectEqual(live_after_load, engine.dead_bytes);
try testing.expect(engine.dead_bytes >= engine.compact_threshold);
try testing.expect(engine.take_compact());
// And the rebuild actually clears the garbage it was called for.
try engine.compact();
try testing.expectEqual(@as(u64, 0), engine.dead_bytes);
try testing.expectEqual(@as(u64, 200), engine.live_docs);
// The engine's live total is the sum over collections, both after a rebuild
// and after the catalog round trip below.
const coll = engine.get_collection("app", "c").?;
try testing.expectEqual(engine.live_bytes, coll.live_bytes);
try testing.expectEqual(coll.slab_used, coll.live_bytes);
}
test "a rebuild leaves the space it reclaimed ready to reuse" {
// Mutation check: delete the second `checkpoint()` at the end of `compact`.
// Red -- one publish only moves the abandoned extents from `pending` to
// `hold`, so nothing is reusable and the next rebuild grows the file instead.
// Measured on the churn gate as ~1.2x of extra steady-state size (3.58x live
// data against 2.47x) under sustained update churn.
var threaded: std.Io.Threaded = .init_single_threaded;
defer threaded.deinit();
var env = test_env(&threaded);
const io = env.io;
const gpa = testing.allocator;
var tmp = try TmpLog.init(gpa);
defer tmp.deinit(gpa);
var engine = try Engine.open(gpa, io, tmp.path);
defer engine.deinit();
engine.compact_threshold = 1024;
try engine.lock();
defer engine.unlock();
for (0..300) |i| {
var d = try make_doc(gpa, @intCast(i), "x");
defer d.deinit();
try engine.insert("app", "c", &d, &env.gen);
}
for (0..300) |i| {
var d = try make_doc(gpa, @intCast(i), "yy");
defer d.deinit();
try engine.replace("app", "c", &d, &env.gen);
}
try engine.commit();
try testing.expect(engine.take_compact());
try engine.compact();
// The rebuild abandoned the old slab extent and every page the old trees
// occupied. Those must be handed back, not merely queued.
try testing.expect(engine.pager.free_ready_pages() > 0);
// And the next allocation actually uses them rather than the tail.
const tail_before = engine.pager.alloc_tail;
for (0..300) |i| {
var d = try make_doc(gpa, @intCast(i), "zzz");
defer d.deinit();
try engine.replace("app", "c", &d, &env.gen);
}
try engine.commit();
try testing.expect(engine.pager.alloc_tail < tail_before + engine.pager.free_ready_pages() + 64);
try testing.expectEqual(@as(u64, 300), engine.live_docs);
}
test "reopen carries the garbage counter across a restart" {
var threaded: std.Io.Threaded = .init_single_threaded;
defer threaded.deinit();
var env = test_env(&threaded);
const io = env.io;
const gpa = testing.allocator;
var tmp = try TmpLog.init(gpa);
defer tmp.deinit(gpa);
var dead_before: u64 = 0;
var live_before: u64 = 0;
{
var engine = try Engine.open(gpa, io, tmp.path);
defer engine.deinit();
engine.compact_threshold = std.math.maxInt(u64); // never rebuild here
try engine.lock();
for (0..100) |i| {
var d = try make_doc(gpa, @intCast(i), "x");
defer d.deinit();
try engine.insert("app", "c", &d, &env.gen);
}
for (0..50) |i| {
var d = try make_doc(gpa, @intCast(i), "yy");
defer d.deinit();
try engine.replace("app", "c", &d, &env.gen);
}
try engine.commit();
try engine.checkpoint();
dead_before = engine.dead_bytes;
live_before = engine.live_bytes;
engine.unlock();
try testing.expect(dead_before > 0);
}
var engine2 = try Engine.open(gpa, io, tmp.path);
defer engine2.deinit();
try testing.expectEqual(dead_before, engine2.dead_bytes);
try testing.expectEqual(live_before, engine2.live_bytes);
}
test "reopen replays log" {
var threaded: std.Io.Threaded = .init_single_threaded;
defer threaded.deinit();