# Fuzz / torture harnesses

Black-box and in-process harnesses that hunt for engine bugs and bottlenecks
that the fixed e2e scenarios cannot reach. The e2e suites prove specific
behaviours; these generate their own workloads and check invariants.

## crash-fuzz.js — crash-consistency fuzzer

The P0 harness for the M0 (mmap + WAL) crash story. Drives the server through
the official driver with a random write workload (insert batches, single
inserts, `$set`/`$inc`/`$unset` updates, deletes, `createIndex`), kills it with
`SIGKILL` at a random point, reopens the same log file, and verifies the
recovered state against an in-memory model.

**The invariant under test (the "prefix invariant").** With one sequential
client and fsync-before-ack commits, the state that survives a crash must be
`apply(history[0..m))` for some `m` with `acked <= m <= sent`: every
fully-acked write is durable, an in-flight command is either fully there or
fully gone (group commit = one fsync per command), and nothing after it may
survive. The verifier also checks, at the matched prefix:

- the database always opens (replay never refuses — PLAN ground rule 4);
- the `k_1` index exists iff its create record is in the prefix, and
  `find({k: v})` returns exactly the docs with `k == v` (index rebuild after
  replay must be correct, whether the query used the index or a scan);
- `countDocuments({})` matches the model.

Cycles share one log file, so checkpoints, log truncation, COW free-list
reuse and compaction (in `--heavy`) are exercised across cycles. The server
spawned by the harness may never self-crash: a Zig panic or a replay refusal
is reported as a finding with the server log.

**Usage**

```sh
zig build                                   # fresh server binary first!
node tests/fuzz/crash-fuzz.js               # 60 quick cycles, random seed
node tests/fuzz/crash-fuzz.js --seed 42     # reproducible scenario
node tests/fuzz/crash-fuzz.js --heavy       # big batches + 1 MB compact
                                            # threshold: hits checkpoint/
                                            # compaction windows
node tests/fuzz/crash-fuzz.js --no-kill --verify-exec
                                            # graceful stop + reopen, and
                                            # read-back every update (isolates
                                            # execution bugs from replay bugs)
```

Options: `--seed N --rounds N --max-docs N --batch-max N --kill-delay-ms N
--verbose --keep-log --port N`. On failure a repro artifact is written to
`/tmp/crash-fuzz-fail-<seed>.json` and the rerun command is printed.

Determinism: the op sequence and the kill decision come from a seeded PRNG,
so the same seed reproduces the same scenario. Kill timing inside the window
is OS-scheduled (as in any crash tester — RocksDB db_crashtest works the same
way); a failure's seed + artifact reproduce the *scenario*, and the invariant
check is timing-independent.

**Mutation-checked** (repo ground rule 2): making the verifier require
`m == sent` (over-strict) turns seeds with lost in-flight ops red, proving the
harness actually observes both prefix states (`acked` and `sent`), not just
the full state.

## Notes on what the crash fuzzer does *not* cover (by design)

- **Torn log tails / torn watermark writes**: `kill -9` is process death, not
  power loss — page cache survives, so a partially-written block is rare and
  OS-scheduled. The replay fuzzer (log truncation at random offsets) is the
  deterministic way to hit that surface; it is planned (P0 item 2) but not
  yet built.
- **Concurrent clients**: the fuzzer uses one sequential client so the prefix
  invariant is exactly checkable. Race/interleaving coverage stays with
  `tests/e2e/e2e2.js concurrent`.
- **Semantic differential vs real MongoDB**: planned (P1 item 4).

## Known engine observations (findings so far)

- A small churn-heavy database can exhaust the pager's 64 GB address-space
  reservation: the data file grows in ≥8 MiB steps, compounding, and never
  shrinks until a compaction (data-file rebuild) or checkpoint reclaims it.
  With `--compact-threshold` left at the 16 MB default and a small log, the
  file can reach 64 GB and writes start failing with `DatabaseTooLarge` after
  enough growth events. `--heavy` passes `--compact-threshold 1 MiB` to keep
  the file bounded and to fuzz the rebuild+crash path.