# AGENTS.md — operating guide for AI agents working in this repo Read this first. It exists so any agent — in any session, without the original conversation's context — can work here safely and verifiably. ## What this is MultiforaDB: a lightweight, embedded, MongoDB-compatible document database in Zig 0.16. It speaks the MongoDB wire protocol (OP_MSG, maxWireVersion 8) so real drivers (`mongosh`, Node, PyMongo) connect over TCP. Everything lives in a single-file LZ4-block log with an in-memory engine, B+tree indexes, and TTL/unique/sparse/compound index support. **Forward plan**: the project's direction — a full-fledged embedded, tens-of-GB, maximally MongoDB-compatible database — is decided and written in **[PLAN.md](PLAN.md)**. The current milestone is **M0 (mmap + WAL storage foundation)**. Before starting any work, read PLAN.md; its decision record (D1-D9) and ground rules are binding. ## Read first, in order 1. **PLAN.md** — the plan: decisions, milestones M0-M9, gates, work breakdown, deferred designs. The next session always starts here. 2. **README.md** — current state, features, benchmark table, known gaps, working-with-large-collections notes. 3. **ROADMAP.md** — design write-ups of the completed performance work (B+tree, ordered `_id` index, compressed log, byte storage, decomposed locks). Contains the project's ground rules and verification recipes. ## Toolchain - **Zig 0.16.0** — pinned. Do not assume a newer Zig is compatible. - **Node + the official `mongodb` driver** (in `tests/e2e/node_modules`) for e2e. `npm install` in `tests/e2e` if node_modules is missing. ## Build / test / run ```sh zig build # builds zig-out/bin/multiforadb (ReleaseFast by default) zig build test # unit tests, ReleaseFast zig build test -Doptimize=ReleaseSafe # unit tests with safety checks on zig-out/bin/multiforadb --port 27017 --db data.log ``` **Critical traps:** - **`zig build test` does NOT refresh the server binary.** After any change under `src/`, rebuild with `zig build` before restarting the server, or e2e will run against stale code and report failures that the source no longer explains. (This has bitten repeatedly; it is documented in `tests/e2e/README.md`.) - **Debug builds are 10-200x slower** than ReleaseFast on every path. Never benchmark (or time out waiting for) a Debug server. `zig build` defaults to ReleaseFast on purpose. ## E2E discipline (any change to src/) ```sh zig build # rebuild the server first! zig build test # ReleaseFast zig build test -Doptimize=ReleaseSafe ``` Then, against a server on port 27020 (TTL suites need `--ttl-sweep-secs 1`): ```sh node tests/e2e/e2e.js # CRUD + operators + aggregate + errors node tests/e2e/e2e2.js concurrent node tests/e2e/e2e2.js crash-a # write 50 docs, kill -9 the server node tests/e2e/e2e2.js crash-b # restart, verify all 50 survived node tests/e2e/e2e3.js # secondary indexes node tests/e2e/e2e4.js # TTL indexes (server must run --ttl-sweep-secs 1) node tests/e2e/e2e6.js # self-contained full lifecycle (spawns its own server, incl. kill -9) ``` Which suites to run for a given change: - anything touching the write path or log format → the crash pair (e2e2 crash-a/b) and e2e6 - anything touching indexes → e2e3.js and e2e4.js - everything → all of the above `tests/e2e/README.md` has the full matrix, ports, and harness docs (`big.js` for multi-GB loads, `compare-run.sh`/`bench-run.sh` for the benchmark gate). ## Spec tests (the compatibility gate — PLAN D2) ```sh bash tests/spec/fetch.sh # pinned mongodb/specifications suites node tests/spec/run.js --scorecard # run + re-record tests/spec/scorecard.txt node tests/spec/run.js --file find.json --verbose ``` `tests/spec/scorecard.txt` is committed and re-recorded every milestone (PLAN D9), so progress is verifiable across sessions. It is a *measuring* tool: a red baseline is the expected state at M0, `run.js` exits 0 as long as it ran, and nothing unimplemented is ever counted as a pass. `tests/spec/README.md` has the scope, the pins, and what is not yet asserted. ## Ground rules (binding — from ROADMAP.md and PLAN.md) 1. **Measure A/B on one harness. Do not trust the model.** Several predictions made during the existing work were wrong in both directions. The cheap A/B is: flip one line, rebuild, run, flip it back. 2. **Mutation-check any test guarding an invariant.** Break the thing the test is supposed to catch and confirm it goes red before trusting it. 3. **The index invariant is absolute** (`src/index.zig` header): an index only *generates candidates*; the full filter is re-applied afterwards. Over-approximating is slow. Under-approximating is a wrong answer. 4. **The database must always open.** Replay never refuses to start over recoverable damage (e.g., a unique index finding duplicates in existing data warns and keeps going). 5. **The checkpoint never describes a state ahead of the durable log tail** (M0 invariant): watermark advancement is the last step of the checkpoint protocol, after the data-file fsync. 6. **No page below the stable mark is ever stored into** (M0, PLAN Amendment A1). The whole crash story is downstream of this one rule: a mutation that would dirty a page belonging to the last durable checkpoint copies it to a fresh page first. Enforced structurally by a copy-on-write page accessor, and mechanically by `mprotect`-ing the stable prefix read-only in ReleaseSafe/test builds — a missed COW then segfaults in the suite instead of corrupting a database silently. 7. **A checkpoint never renumbers slab offsets** (M0, Amendment A3). Index leaves hold physical document offsets; only a full rebuild may move documents, and it rebuilds every index in the same pass. 8. **Write-then-extend discipline for mmap**: ftruncate before touching new pages; never fault past the end of the mapped file (SIGBUS). 9. **`Engine.seq` is restored on open** as `max(watermark.seq, max replayed record.seq)`, and a log rewrite never emits a record whose seq is ≤ the watermark. Today `seq` restarts at 0 on every open — harmless without a watermark, silent data loss with one. Note for anyone reading an older revision: rule 6 previously read "index entries own their key bytes". That was a misdiagnosis — `store_record` already copies keys. See PLAN Amendment A3 for what the actual trap was. ## Code style Binding reference: **`docs/TIGER_STYLE.md`** (TigerBeetle's TigerStyle, adopted verbatim). Read it before writing code. Its mechanical rules are enforced: `zig fmt` clean, 4-space indent, hard 100-column line limit, no `if` without braces unless it fits on one line, snake_case for functions/ variables/files, units last in names (`latency_ms_max`), and always-on assertions (the repo's `assert.zig`, see below). Project-specific rules and deliberate deviations from TigerStyle: - Zig 0.16 idioms: `std.Io` threaded through everything, unmanaged containers, arena-threaded allocators where the code does. - House style: user-declared functions use `snake_case`; see existing `src/` for the pattern. Keep `src/lib.zig` as the library root and keep the server (`src/server.zig`) a thin front-end — the C API seam (PLAN D1) depends on that staying clean. - Public code invariants are asserted with the repo's own `assert.zig` (always active, including ReleaseFast) rather than `std.debug.assert`, so they do not silently vanish in optimized builds. - **Deviation (allocation):** TigerStyle's "statically allocated at startup, no allocation after init" is not yet this repo's architecture — the engine is arena-threaded. Do not introduce new *long-lived* dynamic allocations on hot paths; arenas scoped to a request are fine. Static allocation is a deferred design item (PLAN). - **Deviation (recursion):** TigerStyle bans recursion. `bson.zig`'s document parser is still recursive (nested docs/arrays); keep new code iterative and do not extend the recursion depth usage without a design note. Bounding parser depth is tracked work. - **Functions stay under 70 lines** (TigerStyle hard limit). A handful of pre-existing offenders remain (see the reflow/refactor backlog in ROADMAP notes); new code must not add to the count. ## Source map ## Source map ``` src/bson.zig BSON parse/serialize, ObjectId, canonical comparison/key encoding src/wire.zig OP_MSG/OP_QUERY framing, message + reply builders src/commands.zig command dispatch (hello, CRUD, aggregate, admin, indexes) src/server.zig TCP accept loop, per-connection handlers, TTL sweep monitor src/db.zig engine: db → collection → _id → document maps, slab storage src/storage.zig append-only log: blocks, LZ4, XxHash3, replay, compaction src/query.zig filter matcher, regex engine, sort, projection src/index.zig B+tree indexes: entries, search, query planner src/update.zig update operators with dot-path navigation src/main.zig CLI: --port, --bind, --db, --ttl-sweep-secs, --compact-threshold ``` ## Working on a milestone 1. Re-read PLAN.md (and this file). Confirm which milestone is current. 2. Per-milestone gates are in PLAN.md; the M0 gate is 6 items ending with benchmark parity (phase8 baseline) and the spec-test runner red baseline. 3. Commit scorecard and benchmark results with each milestone (PLAN D9) so progress stays verifiable across sessions. 4. Deferred designs (cursors, aggregation, transactions, change streams, C API) are deliberately *not* specified yet — grill the design with the human before implementing (PLAN section 6).