Aleksey Shakhmatov
cd88e1a4d1
Two bugs, both of which the crash fuzzer surfaced and neither of which any
existing test could see.
**A split put the new sibling in the wrong slot when separators repeat.**
`split_leaf` located the new right sibling with `separator_pos(node, key)`, a
search for the promoted key. That agrees with "immediately after `left`" only
while separators are distinct. When several children share one -- ten distinct
values across thousands of documents, so each value spans dozens of leaves --
`separator_pos` returns the slot after the *whole* equal-key run, which puts
the sibling at the end of that run while the leaf chain has it right after
`left`.
Parent child order then stops matching leaf chain order, and that is the one
thing a lookup cannot survive: `descend_lower` picks the last child of the equal
run, and `lookup_eq` walks forward from there over keys *smaller* than the one
it wants, stops at the first mismatch, and reports nothing. Every entry is
present, the chain is correctly ordered, `count()` is right -- and the query
returns empty. `crash-fuzz.js` found it after ~700 heavy cycles as
`find({k: 3})` returning 0 of 401 documents while every other key was exact.
Fixed by `child_slot_after`, which is positional by construction.
**mmap growth rounded with a non-power-of-two alignment.** Past 64 MiB the
growth chunk becomes a proportion of the current size (`mapped_pages / 8`),
which is not a power of two -- and `std.mem.alignForward` asserts that it is.
In safe builds that panicked; in ReleaseFast, where the assert is compiled out,
it computed `(addr + align - 1) & ~(align - 1)` with a non-power-of-two mask,
which can round *down*. A mapping shorter than intended is survivable, but a
mapping longer than the file is exactly what this function exists to prevent: a
store into a mapped page past end-of-file raises SIGBUS, which no error path
catches. `alignForwardAnyAlign` instead. Never noticed because no unit test grew
a pager past 64 MiB.
Also here, because both bugs were invisible rather than merely unfixed:
- `assert_indexes_cover_every_document` (db.zig) checks the index invariant
directly -- an index generates candidates and the full filter is re-applied to
those, so a missing entry is a missing query result nothing else detects.
- `Index.unreachable_key_count` counts keys present in the leaf chain but not
reachable by descending from the root, which is precisely the state above:
healthy by every other measure.
- `Index.dbg_root` dumps parent/chain agreement. Marked TEMPORARY; drop it once
the invariant checks have earned their keep.
- `crash-fuzz.js` now asks the same question without the index, so a failure
says whether the documents are wrong or only the index's answer about them,
and reports per-key totals so one lost leaf is distinguishable from an empty
index.
Verified: `zig build test` in ReleaseFast and ReleaseSafe, and seeded fuzzer
runs that previously reproduced the split bug.
Fuzz / torture harnesses
Black-box and in-process harnesses that hunt for engine bugs and bottlenecks that the fixed e2e scenarios cannot reach. The e2e suites prove specific behaviours; these generate their own workloads and check invariants.
crash-fuzz.js — crash-consistency fuzzer
The P0 harness for the M0 (mmap + WAL) crash story. Drives the server through
the official driver with a random write workload (insert batches, single
inserts, $set/$inc/$unset updates, deletes, createIndex), kills it with
SIGKILL at a random point, reopens the same log file, and verifies the
recovered state against an in-memory model.
The invariant under test (the "prefix invariant"). With one sequential
client and fsync-before-ack commits, the state that survives a crash must be
apply(history[0..m)) for some m with acked <= m <= sent: every
fully-acked write is durable, an in-flight command is either fully there or
fully gone (group commit = one fsync per command), and nothing after it may
survive. The verifier also checks, at the matched prefix:
- the database always opens (replay never refuses — PLAN ground rule 4);
- the
k_1index exists iff its create record is in the prefix, andfind({k: v})returns exactly the docs withk == v(index rebuild after replay must be correct, whether the query used the index or a scan); countDocuments({})matches the model.
Cycles share one log file, so checkpoints, log truncation, COW free-list
reuse and compaction (in --heavy) are exercised across cycles. The server
spawned by the harness may never self-crash: a Zig panic or a replay refusal
is reported as a finding with the server log.
Usage
zig build # fresh server binary first!
node tests/fuzz/crash-fuzz.js # 60 quick cycles, random seed
node tests/fuzz/crash-fuzz.js --seed 42 # reproducible scenario
node tests/fuzz/crash-fuzz.js --heavy # big batches + 1 MB compact
# threshold: hits checkpoint/
# compaction windows
node tests/fuzz/crash-fuzz.js --no-kill --verify-exec
# graceful stop + reopen, and
# read-back every update (isolates
# execution bugs from replay bugs)
Options: --seed N --rounds N --max-docs N --batch-max N --kill-delay-ms N --verbose --keep-log --port N. On failure a repro artifact is written to
/tmp/crash-fuzz-fail-<seed>.json and the rerun command is printed.
Determinism: the op sequence and the kill decision come from a seeded PRNG, so the same seed reproduces the same scenario. Kill timing inside the window is OS-scheduled (as in any crash tester — RocksDB db_crashtest works the same way); a failure's seed + artifact reproduce the scenario, and the invariant check is timing-independent.
Mutation-checked (repo ground rule 2): making the verifier require
m == sent (over-strict) turns seeds with lost in-flight ops red, proving the
harness actually observes both prefix states (acked and sent), not just
the full state.
Notes on what the crash fuzzer does not cover (by design)
- Torn log tails / torn watermark writes:
kill -9is process death, not power loss — page cache survives, so a partially-written block is rare and OS-scheduled. The replay fuzzer (log truncation at random offsets) is the deterministic way to hit that surface; it is planned (P0 item 2) but not yet built. - Concurrent clients: the fuzzer uses one sequential client so the prefix
invariant is exactly checkable. Race/interleaving coverage stays with
tests/e2e/e2e2.js concurrent. - Semantic differential vs real MongoDB: planned (P1 item 4).
Known engine observations (findings so far)
- A small churn-heavy database can exhaust the pager's 64 GB address-space
reservation: the data file grows in ≥8 MiB steps, compounding, and never
shrinks until a compaction (data-file rebuild) or checkpoint reclaims it.
With
--compact-thresholdleft at the 16 MB default and a small log, the file can reach 64 GB and writes start failing withDatabaseTooLargeafter enough growth events.--heavypasses--compact-threshold 1 MiBto keep the file bounded and to fuzz the rebuild+crash path.