Files

Aleksey Shakhmatov a9f625f82a tests/fuzz: listIndexes on a namespace the prefix never created is expected

`crash-fuzz.js` aborted with "harness error: MongoServerError: ns not found"
whenever the surviving prefix contained no write that created the collection --
a kill during the first in-flight command on a fresh log. `listIndexes` on a
missing namespace is NamespaceNotFound, which is what real MongoDB answers too,
so the server was right and the harness treated a legitimate state as its own
failure. Worse, it aborted the run instead of verifying that state, which is
exactly the state worth verifying.

Reproduces with `--seed 1234 --rounds 60` and is why seeded runs were unusable;
`--heavy` happened to miss it. Confirmed against the previous commit before
changing anything, so it is the harness and not the engine.

Both seeds now pass 60 cycles.

2026-08-04 00:41:03 +03:00

crash-fuzz.js

tests/fuzz: listIndexes on a namespace the prefix never created is expected

2026-08-04 00:41:03 +03:00

README.md

tests/fuzz: crash-consistency fuzzer (crash-fuzz.js + README)

2026-08-04 00:05:09 +03:00

README.md

Fuzz / torture harnesses

Black-box and in-process harnesses that hunt for engine bugs and bottlenecks that the fixed e2e scenarios cannot reach. The e2e suites prove specific behaviours; these generate their own workloads and check invariants.

crash-fuzz.js — crash-consistency fuzzer

The P0 harness for the M0 (mmap + WAL) crash story. Drives the server through the official driver with a random write workload (insert batches, single inserts, $set/$inc/$unset updates, deletes, createIndex), kills it with SIGKILL at a random point, reopens the same log file, and verifies the recovered state against an in-memory model.

The invariant under test (the "prefix invariant"). With one sequential client and fsync-before-ack commits, the state that survives a crash must be apply(history[0..m)) for some m with acked <= m <= sent: every fully-acked write is durable, an in-flight command is either fully there or fully gone (group commit = one fsync per command), and nothing after it may survive. The verifier also checks, at the matched prefix:

the database always opens (replay never refuses — PLAN ground rule 4);
the k_1 index exists iff its create record is in the prefix, and find({k: v}) returns exactly the docs with k == v (index rebuild after replay must be correct, whether the query used the index or a scan);
countDocuments({}) matches the model.

Cycles share one log file, so checkpoints, log truncation, COW free-list reuse and compaction (in --heavy) are exercised across cycles. The server spawned by the harness may never self-crash: a Zig panic or a replay refusal is reported as a finding with the server log.

Usage

zig build                                   # fresh server binary first!
node tests/fuzz/crash-fuzz.js               # 60 quick cycles, random seed
node tests/fuzz/crash-fuzz.js --seed 42     # reproducible scenario
node tests/fuzz/crash-fuzz.js --heavy       # big batches + 1 MB compact
                                            # threshold: hits checkpoint/
                                            # compaction windows
node tests/fuzz/crash-fuzz.js --no-kill --verify-exec
                                            # graceful stop + reopen, and
                                            # read-back every update (isolates
                                            # execution bugs from replay bugs)

Options: --seed N --rounds N --max-docs N --batch-max N --kill-delay-ms N --verbose --keep-log --port N. On failure a repro artifact is written to /tmp/crash-fuzz-fail-<seed>.json and the rerun command is printed.

Determinism: the op sequence and the kill decision come from a seeded PRNG, so the same seed reproduces the same scenario. Kill timing inside the window is OS-scheduled (as in any crash tester — RocksDB db_crashtest works the same way); a failure's seed + artifact reproduce the scenario, and the invariant check is timing-independent.

Mutation-checked (repo ground rule 2): making the verifier require m == sent (over-strict) turns seeds with lost in-flight ops red, proving the harness actually observes both prefix states (acked and sent), not just the full state.

Notes on what the crash fuzzer does not cover (by design)

Torn log tails / torn watermark writes: kill -9 is process death, not power loss — page cache survives, so a partially-written block is rare and OS-scheduled. The replay fuzzer (log truncation at random offsets) is the deterministic way to hit that surface; it is planned (P0 item 2) but not yet built.
Concurrent clients: the fuzzer uses one sequential client so the prefix invariant is exactly checkable. Race/interleaving coverage stays with tests/e2e/e2e2.js concurrent.
Semantic differential vs real MongoDB: planned (P1 item 4).

Known engine observations (findings so far)

A small churn-heavy database can exhaust the pager's 64 GB address-space reservation: the data file grows in ≥8 MiB steps, compounding, and never shrinks until a compaction (data-file rebuild) or checkpoint reclaims it. With --compact-threshold left at the 16 MB default and a small log, the file can reach 64 GB and writes start failing with DatabaseTooLarge after enough growth events. --heavy passes --compact-threshold 1 MiB to keep the file bounded and to fuzz the rebuild+crash path.