pager: reuse the data file when there is no checkpoint to honour
`Pager.open` set `alloc_tail` to the end of the existing file -- "everything already in the file is allocated until a watermark narrows it down". Safe when a watermark exists. When one does not, it is the opposite of safe: nothing in the file is referenced, the log is the whole truth and replay is about to rebuild the slab, the trees and the overflow from it, so every reopen started allocating *above* the previous copy. With no watermark there is also no free list, so the old copy was never given back. Linear growth per open, unbounded. It does not need a crash. A database small enough never to reach the 32 MB checkpoint threshold never publishes a watermark at all, so *every* clean reopen took this path: 20 documents inserted per cycle, 12 reopen cycles before 17, 34, 50, 67, 85, 102, 118, 135, 168, 201, 236, 269 MB after 17 MB, flat 240 documents in a 269 MB file, heading for `DatabaseTooLarge`. The crash fuzzer shows the same thing under a real workload -- 60 crash/reopen cycles with ~460 documents ended at 2735 MB before, 17 MB after, with the prefix invariant holding either way. That number was sitting in its own output as `data=2735MB` and reads as normal until you divide it by the document count. The file is deliberately not truncated. The mapping already covers these pages and `grow_to` extends the file only when the mapping is too small, so shortening the file behind a mapping that still spans it would turn a later write into SIGBUS. Reusing from the front is what the unbounded growth needed; giving the disk back is a separate change to the same function. Mutation: leave `alloc_tail` at the file end -- red on the new test, which opens, writes and closes three times without a checkpoint and requires the third tail to be within one slab extent of the first.
This commit is contained in:
@@ -352,6 +352,30 @@ pub const Pager = struct {
|
||||
// narrows it down.
|
||||
self.alloc_tail = @max(page_first_data, self.file_pages);
|
||||
try self.load_watermark();
|
||||
if (self.loaded.generation == 0) {
|
||||
// No usable checkpoint: either none was ever published, or both
|
||||
// watermark slots were unreadable. Then *nothing* in this file is
|
||||
// referenced -- the log is the whole truth and replay is about to
|
||||
// rebuild the slab, the trees and the overflow from it -- so the
|
||||
// file is free space, not allocated space.
|
||||
//
|
||||
// Leaving `alloc_tail` at the file end instead made every reopen
|
||||
// append a fresh copy above the last one, with no watermark and
|
||||
// therefore no free list to ever give the old copy back. Linear
|
||||
// growth per open, unbounded, and it does not need a crash: a
|
||||
// database small enough never to reach the checkpoint threshold
|
||||
// never publishes a watermark at all, so *every* clean reopen
|
||||
// took this path. Measured at 20 documents per cycle: +17 MB per
|
||||
// reopen, 269 MB after twelve, on course for DatabaseTooLarge.
|
||||
//
|
||||
// The file is not truncated here on purpose. The mapping is
|
||||
// already established over these pages, and `grow_to` extends the
|
||||
// file only when the mapping is too small -- so shortening the
|
||||
// file behind a mapping that still covers it would turn a later
|
||||
// write into SIGBUS. Reusing from the front bounds the file at
|
||||
// its high-water mark, which is what the unbounded growth needed.
|
||||
self.alloc_tail = page_first_data;
|
||||
}
|
||||
}
|
||||
return self;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user