`slab_used` only ever grew by a document's length, so the two places the
appender writes slab off went uncounted: the gap left when a checkpoint
freezes the page the cursor points into and the cursor resumes at the next
system page, and the tail of an extent abandoned for a document that no
longer fits. Both are real garbage -- only a rebuild gets them back -- and
both were invisible to the trigger that decides whether a rebuild is worth
doing. An abandoned tail can be most of 8 MiB.
`note_skip` counts them into `slab_used` where they happen and hands the
number back for the caller to charge to the engine, which keeps the identity
the last commit established: `dead_bytes` is the sum of
`slab_used - live_bytes` over the collections that exist.
That identity is also why `compact` no longer zeroes `dead_bytes`. A repack
appends through the same slab, so it abandons a tail of its own whenever the
next document does not fit; zeroing was true only if a rebuild leaves nothing
behind, and it does not. `sum_dead_bytes` recomputes it from the collections,
each under its own lock.
Carried with it, because this commit is what exposed it: the four engine
counters get a lock of their own. They are the only engine-wide mutable state
a writer touches while holding nothing but its own collection's lock, so two
writers on different collections reach them with no lock in common -- and the
checkpoint's consistency check read them ordered against nothing, while the
per-collection figures it compares them to were read under each collection's
lock. Before this commit `dead_bytes` moved on nearly every write, so that
check was skipped almost every time; with skips counted it stands still
between checkpoints, the check runs, and it aborted three of eight ReleaseSafe
runs of "a checkpoint runs alongside writers on several collections".
Not mutation-checked, and worth saying so: reverting the lock did not
re-trigger the abort in 24 further runs, and neither did the exact pre-fix
revision in 10. The rate depends on machine load, and a mutation check that
cannot be relied on to go red is not a check. The lock stands on inspection
instead -- an unsynchronized read-modify-write on a counter shared by threads
holding no common lock is a defect whatever its rate -- and the concurrency
test now asserts the identity once everything is quiet, which is the half of
it that does not depend on a race being caught in the act.
Mutation-checked, each red on its own: drop either `note_skip` call in
`slab_reserve`; put `self.dead_bytes = 0;` back in `compact`. The `note_skip`
in `slab_append` is covered by the concurrency test, the only place a publish
lands between a reservation and its append.
165/165 unit tests in ReleaseFast and ReleaseSafe, 82/82 fuzz, e2e 49,
e2e2 concurrent 2 + crash pair, e2e3 16, e2e4 17, e2e6 72, e2e7 86,
crash-fuzz 60 cycles.
332206e5dd