319a515b89fdc383ef6b60ca2f38b37743edc3fc
9 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| 319a515b89 |
pager: reuse the data file when there is no checkpoint to honour
`Pager.open` set `alloc_tail` to the end of the existing file -- "everything already in the file is allocated until a watermark narrows it down". Safe when a watermark exists. When one does not, it is the opposite of safe: nothing in the file is referenced, the log is the whole truth and replay is about to rebuild the slab, the trees and the overflow from it, so every reopen started allocating *above* the previous copy. With no watermark there is also no free list, so the old copy was never given back. Linear growth per open, unbounded. It does not need a crash. A database small enough never to reach the 32 MB checkpoint threshold never publishes a watermark at all, so *every* clean reopen took this path: 20 documents inserted per cycle, 12 reopen cycles before 17, 34, 50, 67, 85, 102, 118, 135, 168, 201, 236, 269 MB after 17 MB, flat 240 documents in a 269 MB file, heading for `DatabaseTooLarge`. The crash fuzzer shows the same thing under a real workload -- 60 crash/reopen cycles with ~460 documents ended at 2735 MB before, 17 MB after, with the prefix invariant holding either way. That number was sitting in its own output as `data=2735MB` and reads as normal until you divide it by the document count. The file is deliberately not truncated. The mapping already covers these pages and `grow_to` extends the file only when the mapping is too small, so shortening the file behind a mapping that still spans it would turn a later write into SIGBUS. Reusing from the front is what the unbounded growth needed; giving the disk back is a separate change to the same function. Mutation: leave `alloc_tail` at the file end -- red on the new test, which opens, writes and closes three times without a checkpoint and requires the third tail to be within one slab extent of the first. |
|||
| 1814020df9 |
db/pager: an append resumes inside its extent after a checkpoint
`slab_reserve` and `reserve_overflow` abandoned the rest of their extent whenever a checkpoint froze the page the tail pointed into, and took a fresh 8 MiB one. The comment called the waste "bounded by one extent per collection per checkpoint", which is true per checkpoint and says nothing about the sum: nothing reclaims it except a rebuild, and a rebuild only runs when there is garbage. A pure-insert workload produces none. Measured, 40 collections of inserts with incompressible payloads so the log actually reaches the checkpoint threshold: live data file log 29 MB 340 MB 29 MB 38 MB 542 MB 5 MB <- checkpoint 67 MB 681 MB 33 MB 76 MB 1076 MB 9 MB <- checkpoint 115 MB 1357 MB 14 MB <- checkpoint 11.8x the live data and climbing by ~335 MB per checkpoint (40 x 8 MiB), which would exhaust the 64 GB address-space reservation after roughly 6 GB of real data -- and after ~1.2 GB with 200 collections. `DatabaseTooLarge` on a database that is nowhere near too large. The fix is what the plan called for and never got: round the cursor up to the next *system* page and keep the extent. Only the page holding the live tail is in the published image; the rest of the extent holds nothing referenced by the image or by an index, so `Pager.mark_appendable` hands it back for appending (and unprotects it, since it may sit below the stable mark where `protect_image` made it read-only). System pages rather than 4 KiB ones because writeback tears at the granularity the kernel manages: a 4 KiB store dirties a whole 16 KiB page on Apple Silicon, and tearing there would take out the published bytes sharing it. Same 40 collections after: 340 MB -> 352 MB across three checkpoints, the ratio falling monotonically toward the 8 MiB-per-collection floor. 64,000 documents across 8 collections verified byte-for-byte and after a kill -9. The churn gate is unchanged at 1.65x, big.js at 4 GB unchanged (4.32 GB file, reopen 0.5 s, RSS after reopen 130 MB). Two mutations, verified red: dropping the resume branch (a fresh extent per checkpoint), and rounding to `page_size` instead of `map_align` (the resumed append then shares a system page with the published image). |
|||
| 4b70ce6da9 |
pager: a page reservation belongs to its consumer, not to the pager
The promise `reserve_pages` makes was a single counter on the pager, and the
first concurrent benchmark since the data file landed aborted the server on
it, reliably, at four clients:
assertion failed: page allocation overran reserve_pages' promise
src/index.zig:955 in alloc_node
src/db.zig:794 in upsert
Two upserts on different collections hold different collection locks, so they
run at the same time. Each ends by dropping "whatever is still promised" --
and `release_reservation` zeroed the shared counter, so the first to publish
released the second's promise while the second was still between its log
append and its supposedly infallible allocation. The tripwire fired, which is
the good outcome; the bad one is a growth that never happened and a store past
the mapped end.
This is PLAN risk 3 ("a shared pager makes alloc_tail and free_pending a
global mutex on every insert"), whose mitigation -- private pre-allocated runs
-- was never built. So: `pager.Reservation` is a per-consumer promise, held by
every Index, every Collection (for its doc slab) and the checkpoint, and each
one releases only its own. The pager keeps the sum, which is all `grow_to`
needs. `Engine.release_write_reservations` drops exactly the buckets one
upsert reserved through.
The allocator's own state -- the tail, the total, the free lists, the
unpublished set, file growth -- is now behind `alloc_lock`, taken
uncancelable. It is never held across the log append: that is precisely what
per-consumer reservations buy, and why group commit is unaffected.
concurrent durable insertOne 4 clients 21697 docs/s (was aborting)
16 clients 30678 docs/s
Mutation: make `release_reservation` zero `self.reserved_pages` again. Red on
the new pager test and on three command tests.
|
|||
| 5228ed740a |
db/pager: reclaim what churn abandons
The churn gate (PLAN D6.2 as amended, D7.4) measured a data file growing linearly and without bound: 50% churn over six rounds reached 7.2x the live data and was still climbing when the run was stopped. Three separate bugs, each of which alone was enough to make reclamation impossible. **The compaction trigger had been dead since commit 14.** `note_compact` gated on `log.data_bytes`, which was the right question while the log was the only copy of the data. A checkpoint now truncates the log, and `truncate_to_header` zeroes that counter -- so the first gate stopped being reachable and compaction never fired again. Retarget it at the data file, where the garbage now lives: `Engine.live_bytes`/`dead_bytes`, in bytes rather than document counts because a rewrite copies bytes. The engine's live total is the sum over collections by construction, checked in `write_catalog`, which walks every collection anyway. **`stable_pages` is a bound, not a membership test.** `page_mut_cow` asked `p >= stable_pages`, which is right for tail-bumped pages and wrong for recycled ones -- they come off the free list *below* the mark and are nonetheless writable, because two-generation retention means no live image references them. So every write to a recycled node page copied and freed it again, and both append cursors (the doc slab, the overflow slab) abandoned each recycled extent after a single record. Nothing was ever really reused. Replaced with an exact `unpublished` bit set, cleared at each publish: 32 KiB per GiB, one load against the 4 KiB copy it avoids. **First fit let one-page requests dismantle the extents.** Copy-on-write asks for a single page thousands of times per generation while the doc slab asks for 2048-page extents; first fit carved a page off the front of the largest run every time, so the free list drained to empty every generation with the file still growing by the whole write volume. Best fit keeps the runs whole -- nothing else wants the one-page holes -- and `publish` now coalesces adjacent runs, without which the list only ever fragments. Also: a rebuild publishes twice. One publish moves the abandoned extents from `pending` to `hold`; the space is not reusable until a second, so the next rebuild grew the file instead of reusing what the last one freed. Safe for the reason the delay exists -- what the second publish releases is what the pre-rebuild image referenced, and that image is no longer the fallback. Measured, sustained-churn steady state, 40k x 16 KiB documents: delete half and refill, 6 rounds 4.10x climbing -> 1.65x flat random $set over 5x the collection 3.58x -> 2.47x flat Above the 1.3x the amended D6.2 hoped for, and structurally so: a rebuild needs a whole second copy of the live data before the first can be freed. The gate's purpose was to decide whether doc-level free lists are needed post-M0, and this is the answer -- yes, for M1. Five mutations, each verified red: the numeric mark in `page_mut_cow`, first fit in `take_free`, dropping `coalesce_free_ready`, dropping `mark_unpublished`, and dropping the rebuild's second checkpoint. |
|||
| 138b7f706f |
db/storage: reclaim the log once a checkpoint covers it
The point of a lagging checkpoint: a record whose effect the data file already holds is redundant, so the log can go back to just its header. Without this the log only grows and every open pays for every write ever made. Ordering, which is the whole safety argument: publish the watermark, *then* truncate. The other way round, a crash between them leaves the records gone from the log and absent from any image. A failed truncation is a warning rather than an error -- it costs space and replay time, and loses nothing, so it must not fail a checkpoint that already succeeded. Also wires checkpointing up, which nothing did before. `note_checkpoint` arms it when the log passes a threshold, and the write epilogue and the TTL monitor both claim it -- outside any collection lock, for the same reason compaction runs there: it takes the log lock. The threshold is separate from the compaction one on purpose: compaction is about the garbage share of the data, a checkpoint is about how much replay an open would otherwise do. -- Two things the tests taught me. The first version measured the log before the checkpoint and found 16 bytes -- just the header. Appends buffer in the log's open block and only a commit seals and writes it, so there was nothing on disk to shrink. The test commits first now, and says why. And the "no valid watermark" warning fired for every young database, which is its normal state before the first checkpoint. It now distinguishes a watermark that was *written and cannot be read* from one that was never written -- warning about the ordinary case is how people learn to ignore the warning that matters. Mutation-checked, red: skipping the truncation. Not covered, and the test says so: moving the truncation before the publish, whose failure mode is a crash landing between the two. That needs process-level crash injection, which an in-process test cannot express. |
|||
| d7f7ebb994 |
pager: copy-on-write above the stable mark
The invariant everything else in the crash story rests on (PLAN amendment A1): no page belonging to the last published image is ever stored into, so recovery is `image + replay(seq > watermark)` and the image's bytes are exactly what the watermark described. `page_mut_cow` takes a *pointer to the owner's page number*. That is the load- bearing detail: copy-on-write relocates the page, so the owner has to be told, and a second reference would still aim at the abandoned copy. For the B+tree the owner is the id->page table slot -- which is precisely why node ids are not page numbers. Inert until a checkpoint publishes something, since the stable mark starts at zero. Append-only consumers keep writing in place, except that a checkpoint landing mid-extent freezes the page their tail points into, so both slabs now start a fresh extent rather than writing inside the image. Waste is bounded by one extent per collection per checkpoint. The free list is wired into allocation, which it was not before: copy-on-write abandons every page it touches in every generation, so without reuse the file grows by `generations x touched_set` without bound. That is the difference between a free list being defense-in-depth and being a prerequisite (A2). -- Three things I got wrong on the way, all worth recording. I added a `p >= stable_pages` assert to `page_mut` and had to take it back out. A page recycled off the free list *is* below the mark and *is* legitimately writable -- freed two generations ago, referenced by no live image -- so the page number alone cannot tell a violation from a reuse. The invariant is enforced the two ways A1 actually describes: structurally through `page_mut_cow`, and mechanically through mprotect. The comment says so, since the assert looks like an obvious thing to add. The watermark slots needed a narrow exception, because overwriting the inactive slot is the publication mechanism rather than a violation. It is a separate non-public accessor that asserts its argument is a slot, so it cannot become a general escape hatch. And the mprotect belt: `std.posix.mprotect` does not exist in Zig 0.16, so it is a libc call. It compiled only in ReleaseFast, where the branch is comptime- eliminated -- ReleaseSafe caught that immediately, which is the argument for running both. What the belt's test asserts is that the protection is really applied, not that a violating write faults. A SIGSEGV cannot be caught in-process, and the fault is the OS's behaviour rather than this code's; an mprotect that failed silently would leave a belt that looks present and does nothing, which is the failure worth guarding here. Stated in the test rather than implied. Mutation-checked, all red: COW returning without copying; copying without moving the slot; copying when already above the mark; never reusing a freed page. |
|||
| 2e7f72074f |
index: the node arena and overflow slab live in the data file
The last structures move onto the pager, so the whole engine's storage is now
one mapped file plus the WAL.
Node ids are deliberately *not* page numbers. PLAN amendment A1 explains why:
`Node.parent`, `next`, `prev` and an internal slot's `extra` are back-pointers
by id, so copy-on-write moving a page would force every node referring to it to
move as well -- COWing one leaf cascades through the leaf level, one internal
node through its whole subtree. An in-RAM id->page table makes the table slot
the single owner of a page number, so COW has exactly one pointer to fix. It
costs one dependent load per node access and 4 bytes per node, about 5.6 MB at
100M documents, against the 64-100 bytes *per document* this milestone removes.
The overflow slab becomes extents too, so `Slot.off` for a spilled record is an
absolute file offset -- the same change documents went through.
--
Two bugs, both found by measuring rather than by reading, and both worth
recording because the second one would have been invisible until the churn gate.
The reservation was a tail mark, and it cannot be: an upsert reserves tree pages
for every index *and* slab room for the document, all before one log append. The
second reserver overwrote the first one's promise and the first one's allocation
then asserted. Caught on a 512 MB load by the tripwire added in the
`reserve_for` commit, which is the entire reason that assert exists. It is a
count now, and the multi-consumer ordering is pinned by a test.
And a reservation was never released. It is scoped to one write -- taken before
the log append so the publish cannot fail -- but a tree reservation covers the
worst case of several splits while a typical insert causes none, so the promise
accumulated by a handful of pages per write and dragged the file up with it. The
data file was **1.89 GB for 512 MB of documents**; releasing the unclaimed
promise at the end of each write brings it to 551 MB, or 1.08x, which is the
extent slack and the node pages.
--
Measured on one harness, 512 MB / 16 KB docs, against the in-RAM engine this
replaces:
bulk insert throughput 742.6 MB/s -> 736.4 MB/s
insertOne (sequential) 0.20 ms -> 0.22 ms
createIndex({k: 1}) 26.8 ms -> 16.5 ms
countDocuments({}) 2.1 ms -> 1.1 ms
findOne({k: 500}) indexed 0.75 ms -> 0.56 ms
find({p: range}).count() 6.6 ms -> 4.6 ms
aggregate $group by k 5.8 ms -> 3.8 ms
updateMany({k: 7}, {$inc}) 1.2 ms -> 1.0 ms
Reads gain from one contiguous mapping; the two write rows are within noise of
flat. RSS is still unchanged and still cannot improve, for the reason given in
the previous commit: every open replays the whole log and rebuilds everything.
The dev harnesses each open their own data file now. `zig build fuzz` caught all
four of them, again.
|
|||
| c3e7368477 |
pager: watermark double buffer, page free list, and the ordering that matters
Still engine-unused; this completes the data file's own machinery so the structures can move onto it next. The watermark is what a checkpoint publishes: the log sequence the image covers, the allocation extent, and where the catalog and free list live. Two slots, written by generation parity, so a torn write can never leave zero valid slots -- writing a new generation over the only copy of the old one could. The newer slot whose hash validates wins, and validation happens *before* the generation comparison, or a torn slot wins whenever its garbage generation happens to be larger. A slot that validates is trusted, so it is checked for sense as well as integrity: a rehashed slot with an impossible alloc_tail would otherwise be believed and would describe a file that does not exist. Both slots unreadable opens anyway, with a loud warning and no checkpoint, which means a full replay. Ground rule 4: refusing to start costs more than the warning does. The free list releases pages two generations after they are freed. That is not caution -- it is what keeps generation N-1 a usable image, since its pages stay allocated while N is current, so a torn watermark or a bad catalog can fall back instead of discarding the database. -- Two things worth reading, because both were wrong first. Publish captured alloc_tail *before* writing the free list, so the pages the free list occupies fell outside the published image and the next open would have handed them out again while the watermark still pointed at them. Found by the first test written against it. And the ordering invariant -- every page a watermark describes is durable before the watermark that describes it -- was untestable, which is worse than untested. A kill -9 does not lose page-cache writes, so deleting the sync leaves every test green while making a real power loss unrecoverable. So the pager now records, in test builds only, which pages have been written since the last sync, and a test discards exactly those from the file before reopening. That simulates the one failure a process kill cannot produce. `track_dirty` is `builtin.is_test`, so `page_mut` carries no branch in a real build. Mutation-checked, and the precise result is in the comments because the obvious mutation is not a violation: publish syncs twice before the watermark, so removing either call alone is harmless and correctly stays green. Removing both, or deferring them past the watermark, goes red. Also red: always writing slot A; accepting an impossible alloc_tail; releasing freed pages a generation early. Noted as belt-and-braces rather than claimed as covered: the generation-zero check, which the hash already rejects. |
|||
| 04d56f5b66 |
pager: the data file, its page allocator and the mmap over a fixed reservation
New src/pager.zig, engine-unused at this commit: the structures move onto it in the commits that follow, and landing it alone keeps that change reviewable. The file is an array of 4 KiB pages with a tail-bump extent allocator. PLAN D6.1 asked for a region table; it cannot be one, because there is a node arena per index and a document slab per collection, so the region count is dynamic and unbounded and N contiguous regions cannot all grow at the tail. One page array means exactly one growth path, so the write-then-extend discipline lives in exactly one place, and `ls`/`du` stay honest for the backup story. The mapping is a PROT_NONE anonymous NORESERVE reservation that new file-backed suffixes are mmap'd into with MAP_FIXED. That is one VMA and zero committed pages, and it means **the base never moves for the life of the process**, so a pointer handed out before a growth is still valid after it. The ArrayList-backed arena this replaces could not promise that -- the promoted-key scratch buffer in index.zig exists solely to work around it, and phase8 records a dangling-slab-pointer bug of exactly that shape. Growth is `setLength` and *then* `mmap`, never the reverse: a store into a mapped page past end-of-file raises SIGBUS, which no error path can catch. The accessors assert against `mapped_pages`, so a violation is a panic with a message instead of a signal. `page_mut` is deliberately the only way to obtain a writable page. Copy-on-write hooks in there (commit 12), and funnelling every write through one function is what makes that a change of one body rather than of every caller. `sync` is msync + fsync and only the checkpoint calls it. Between checkpoints dirty pages may sit in the page cache indefinitely, because recovery is `image + replay(seq > watermark)` and the image's pages are never written *differently* -- which is what keeps the write path at exactly one fsync, the WAL's (PLAN amendment A1). This is the one place in src/ that reaches for std.posix, against the house style, and the module comment says why: std.Io.File.MemoryMap prefaults by default, exposes no NORESERVE/FIXED/address hint so it cannot express a reservation, and its setLength is mremap on Linux and unsupported on darwin. Mutation-checked, all four red: remapping the prefix at a kernel-chosen address; dropping the uuid comparison (which is what stops one database's log being replayed onto another's checkpoint); dropping the header hash comparison; and moving setLength after mmap. An empty file is treated as absent rather than as corruption, so a create that died before its header landed still opens. |
