Two bugs, both of which the crash fuzzer surfaced and neither of which any
existing test could see.
**A split put the new sibling in the wrong slot when separators repeat.**
`split_leaf` located the new right sibling with `separator_pos(node, key)`, a
search for the promoted key. That agrees with "immediately after `left`" only
while separators are distinct. When several children share one -- ten distinct
values across thousands of documents, so each value spans dozens of leaves --
`separator_pos` returns the slot after the *whole* equal-key run, which puts
the sibling at the end of that run while the leaf chain has it right after
`left`.
Parent child order then stops matching leaf chain order, and that is the one
thing a lookup cannot survive: `descend_lower` picks the last child of the equal
run, and `lookup_eq` walks forward from there over keys *smaller* than the one
it wants, stops at the first mismatch, and reports nothing. Every entry is
present, the chain is correctly ordered, `count()` is right -- and the query
returns empty. `crash-fuzz.js` found it after ~700 heavy cycles as
`find({k: 3})` returning 0 of 401 documents while every other key was exact.
Fixed by `child_slot_after`, which is positional by construction.
**mmap growth rounded with a non-power-of-two alignment.** Past 64 MiB the
growth chunk becomes a proportion of the current size (`mapped_pages / 8`),
which is not a power of two -- and `std.mem.alignForward` asserts that it is.
In safe builds that panicked; in ReleaseFast, where the assert is compiled out,
it computed `(addr + align - 1) & ~(align - 1)` with a non-power-of-two mask,
which can round *down*. A mapping shorter than intended is survivable, but a
mapping longer than the file is exactly what this function exists to prevent: a
store into a mapped page past end-of-file raises SIGBUS, which no error path
catches. `alignForwardAnyAlign` instead. Never noticed because no unit test grew
a pager past 64 MiB.
Also here, because both bugs were invisible rather than merely unfixed:
- `assert_indexes_cover_every_document` (db.zig) checks the index invariant
directly -- an index generates candidates and the full filter is re-applied to
those, so a missing entry is a missing query result nothing else detects.
- `Index.unreachable_key_count` counts keys present in the leaf chain but not
reachable by descending from the root, which is precisely the state above:
healthy by every other measure.
- `Index.dbg_root` dumps parent/chain agreement. Marked TEMPORARY; drop it once
the invariant checks have earned their keep.
- `crash-fuzz.js` now asks the same question without the index, so a failure
says whether the documents are wrong or only the index's answer about them,
and reports per-key totals so one lost leaf is distinguishable from an empty
index.
Verified: `zig build test` in ReleaseFast and ReleaseSafe, and seeded fuzzer
runs that previously reproduced the split bug.
Comment-only cleanup: the six-line rationale restated the scenario twice
(first-cycle crash at prefix 0 = kill during the first in-flight command
on a fresh log) and echoed 'expected state' with 'exactly the case worth
verifying'. Four lines keep all three points: real MongoDB answers
NamespaceNotFound too, it is expected when nothing durable created the
collection, and treating it as a harness error broke verification of that
case.
`crash-fuzz.js` aborted with "harness error: MongoServerError: ns not found"
whenever the surviving prefix contained no write that created the collection --
a kill during the first in-flight command on a fresh log. `listIndexes` on a
missing namespace is NamespaceNotFound, which is what real MongoDB answers too,
so the server was right and the harness treated a legitimate state as its own
failure. Worse, it aborted the run instead of verifying that state, which is
exactly the state worth verifying.
Reproduces with `--seed 1234 --rounds 60` and is why seeded runs were unusable;
`--heavy` happened to miss it. Confirmed against the previous commit before
changing anything, so it is the harness and not the engine.
Both seeds now pass 60 cycles.
Black-box SIGKILL fuzzer for the M0 mmap+WAL crash story. Random write
workload through the official driver, kill -9 at a random point, reopen the
same log, verify the recovered state against an in-memory model:
- prefix invariant: recovered state == history[0..m) for some m in
[acked, sent]; every acked write durable, in-flight commands all-or-nothing
(group commit), nothing after them may survive
- always-opens (replay never refuses); index presence tied to the prefix and
find({k:v}) correctness (rebuild after replay); countDocuments
- unexpected server death (Zig panic, replay refusal) is a finding with the
server log; --verify-exec read-backs updates to separate execution bugs
from replay bugs; --no-kill for graceful-restart runs; --heavy passes a
1 MiB compact threshold to fuzz compaction/checkpoint windows
Deterministic via seeded PRNG; failures dump a repro artifact with the seed.
Mutation-checked: over-strict prefix check goes red on lost in-flight ops.
Observation recorded: small churn-heavy DBs can exhaust the pager's 64 GB
address-space reservation (data file grows in >=8 MiB compounding steps and
never shrinks without a rebuild), surfacing as DatabaseTooLarge on writes.
