MultiforaDB/tests/fuzz

Fuzz / torture harnesses

Black-box and in-process harnesses that hunt for engine bugs and bottlenecks that the fixed e2e scenarios cannot reach. The e2e suites prove specific behaviours; these generate their own workloads and check invariants.

crash-fuzz.js — crash-consistency fuzzer

The P0 harness for the M0 (mmap + WAL) crash story. Drives the server through the official driver with a random write workload (insert batches, single inserts, $set/$inc/$unset updates, deletes, createIndex), kills it with SIGKILL at a random point, reopens the same log file, and verifies the recovered state against an in-memory model.

The invariant under test (the "prefix invariant"). With one sequential client and fsync-before-ack commits, the state that survives a crash must be apply(history[0..m)) for some m with acked <= m <= sent: every fully-acked write is durable, an in-flight command is either fully there or fully gone (group commit = one fsync per command), and nothing after it may survive. The verifier also checks, at the matched prefix:

• the database always opens (replay never refuses — PLAN ground rule 4);
• the k_1 index exists iff its create record is in the prefix, and find({k: v}) returns exactly the docs with k == v (index rebuild after replay must be correct, whether the query used the index or a scan);
• countDocuments({}) matches the model.

Cycles share one log file, so checkpoints, log truncation, COW free-list reuse and compaction (in --heavy) are exercised across cycles. The server spawned by the harness may never self-crash: a Zig panic or a replay refusal is reported as a finding with the server log.

Usage

zig build                                   # fresh server binary first!
node tests/fuzz/crash-fuzz.js               # 60 quick cycles, random seed
node tests/fuzz/crash-fuzz.js --seed 42     # reproducible scenario
node tests/fuzz/crash-fuzz.js --heavy       # big batches + 1 MB compact
                                            # threshold: hits checkpoint/
                                            # compaction windows
node tests/fuzz/crash-fuzz.js --no-kill --verify-exec
                                            # graceful stop + reopen, and
                                            # read-back every update (isolates
                                            # execution bugs from replay bugs)

Options: --seed N --rounds N --max-docs N --batch-max N --kill-delay-ms N --verbose --keep-log --port N. On failure a repro artifact is written to /tmp/crash-fuzz-fail-<seed>.json and the rerun command is printed.

Determinism: the op sequence and the kill decision come from a seeded PRNG, so the same seed reproduces the same scenario. Kill timing inside the window is OS-scheduled (as in any crash tester — RocksDB db_crashtest works the same way); a failure's seed + artifact reproduce the scenario, and the invariant check is timing-independent.

Mutation-checked (repo ground rule 2): making the verifier require m == sent (over-strict) turns seeds with lost in-flight ops red, proving the harness actually observes both prefix states (acked and sent), not just the full state.

Notes on what the crash fuzzer does not cover (by design)

• Torn log tails / torn watermark writes: kill -9 is process death, not power loss — page cache survives, so a partially-written block is rare and OS-scheduled. The replay fuzzer (log truncation at random offsets) is the deterministic way to hit that surface; it is planned (P0 item 2) but not yet built.
• Concurrent clients: the fuzzer uses one sequential client so the prefix invariant is exactly checkable. Race/interleaving coverage stays with tests/e2e/e2e2.js concurrent.
• Semantic differential vs real MongoDB: planned (P1 item 4).

Known engine observations (findings so far)

• A small churn-heavy database can exhaust the pager's 64 GB address-space reservation: the data file grows in ≥8 MiB steps, compounding, and never shrinks until a compaction (data-file rebuild) or checkpoint reclaims it. With --compact-threshold left at the 16 MB default and a small log, the file can reach 64 GB and writes start failing with DatabaseTooLarge after enough growth events. --heavy passes --compact-threshold 1 MiB to keep the file bounded and to fuzz the rebuild+crash path.