Aleksey Shakhmatov
6c7f1f2e77
Black-box SIGKILL fuzzer for the M0 mmap+WAL crash story. Random write
workload through the official driver, kill -9 at a random point, reopen the
same log, verify the recovered state against an in-memory model:
- prefix invariant: recovered state == history[0..m) for some m in
[acked, sent]; every acked write durable, in-flight commands all-or-nothing
(group commit), nothing after them may survive
- always-opens (replay never refuses); index presence tied to the prefix and
find({k:v}) correctness (rebuild after replay); countDocuments
- unexpected server death (Zig panic, replay refusal) is a finding with the
server log; --verify-exec read-backs updates to separate execution bugs
from replay bugs; --no-kill for graceful-restart runs; --heavy passes a
1 MiB compact threshold to fuzz compaction/checkpoint windows
Deterministic via seeded PRNG; failures dump a repro artifact with the seed.
Mutation-checked: over-strict prefix check goes red on lost in-flight ops.
Observation recorded: small churn-heavy DBs can exhaust the pager's 64 GB
address-space reservation (data file grows in >=8 MiB compounding steps and
never shrinks without a rebuild), surfacing as DatabaseTooLarge on writes.
84 lines
4.3 KiB
Markdown
84 lines
4.3 KiB
Markdown
# Fuzz / torture harnesses
|
|
|
|
Black-box and in-process harnesses that hunt for engine bugs and bottlenecks
|
|
that the fixed e2e scenarios cannot reach. The e2e suites prove specific
|
|
behaviours; these generate their own workloads and check invariants.
|
|
|
|
## crash-fuzz.js — crash-consistency fuzzer
|
|
|
|
The P0 harness for the M0 (mmap + WAL) crash story. Drives the server through
|
|
the official driver with a random write workload (insert batches, single
|
|
inserts, `$set`/`$inc`/`$unset` updates, deletes, `createIndex`), kills it with
|
|
`SIGKILL` at a random point, reopens the same log file, and verifies the
|
|
recovered state against an in-memory model.
|
|
|
|
**The invariant under test (the "prefix invariant").** With one sequential
|
|
client and fsync-before-ack commits, the state that survives a crash must be
|
|
`apply(history[0..m))` for some `m` with `acked <= m <= sent`: every
|
|
fully-acked write is durable, an in-flight command is either fully there or
|
|
fully gone (group commit = one fsync per command), and nothing after it may
|
|
survive. The verifier also checks, at the matched prefix:
|
|
|
|
- the database always opens (replay never refuses — PLAN ground rule 4);
|
|
- the `k_1` index exists iff its create record is in the prefix, and
|
|
`find({k: v})` returns exactly the docs with `k == v` (index rebuild after
|
|
replay must be correct, whether the query used the index or a scan);
|
|
- `countDocuments({})` matches the model.
|
|
|
|
Cycles share one log file, so checkpoints, log truncation, COW free-list
|
|
reuse and compaction (in `--heavy`) are exercised across cycles. The server
|
|
spawned by the harness may never self-crash: a Zig panic or a replay refusal
|
|
is reported as a finding with the server log.
|
|
|
|
**Usage**
|
|
|
|
```sh
|
|
zig build # fresh server binary first!
|
|
node tests/fuzz/crash-fuzz.js # 60 quick cycles, random seed
|
|
node tests/fuzz/crash-fuzz.js --seed 42 # reproducible scenario
|
|
node tests/fuzz/crash-fuzz.js --heavy # big batches + 1 MB compact
|
|
# threshold: hits checkpoint/
|
|
# compaction windows
|
|
node tests/fuzz/crash-fuzz.js --no-kill --verify-exec
|
|
# graceful stop + reopen, and
|
|
# read-back every update (isolates
|
|
# execution bugs from replay bugs)
|
|
```
|
|
|
|
Options: `--seed N --rounds N --max-docs N --batch-max N --kill-delay-ms N
|
|
--verbose --keep-log --port N`. On failure a repro artifact is written to
|
|
`/tmp/crash-fuzz-fail-<seed>.json` and the rerun command is printed.
|
|
|
|
Determinism: the op sequence and the kill decision come from a seeded PRNG,
|
|
so the same seed reproduces the same scenario. Kill timing inside the window
|
|
is OS-scheduled (as in any crash tester — RocksDB db_crashtest works the same
|
|
way); a failure's seed + artifact reproduce the *scenario*, and the invariant
|
|
check is timing-independent.
|
|
|
|
**Mutation-checked** (repo ground rule 2): making the verifier require
|
|
`m == sent` (over-strict) turns seeds with lost in-flight ops red, proving the
|
|
harness actually observes both prefix states (`acked` and `sent`), not just
|
|
the full state.
|
|
|
|
## Notes on what the crash fuzzer does *not* cover (by design)
|
|
|
|
- **Torn log tails / torn watermark writes**: `kill -9` is process death, not
|
|
power loss — page cache survives, so a partially-written block is rare and
|
|
OS-scheduled. The replay fuzzer (log truncation at random offsets) is the
|
|
deterministic way to hit that surface; it is planned (P0 item 2) but not
|
|
yet built.
|
|
- **Concurrent clients**: the fuzzer uses one sequential client so the prefix
|
|
invariant is exactly checkable. Race/interleaving coverage stays with
|
|
`tests/e2e/e2e2.js concurrent`.
|
|
- **Semantic differential vs real MongoDB**: planned (P1 item 4).
|
|
|
|
## Known engine observations (findings so far)
|
|
|
|
- A small churn-heavy database can exhaust the pager's 64 GB address-space
|
|
reservation: the data file grows in ≥8 MiB steps, compounding, and never
|
|
shrinks until a compaction (data-file rebuild) or checkpoint reclaims it.
|
|
With `--compact-threshold` left at the 16 MB default and a small log, the
|
|
file can reach 64 GB and writes start failing with `DatabaseTooLarge` after
|
|
enough growth events. `--heavy` passes `--compact-threshold 1 MiB` to keep
|
|
the file bounded and to fuzz the rebuild+crash path.
|