plan: amend the M0 decision record for copy-on-write; add AGENTS.md
The M0 implementation review found three of D1-D9 wrong or incomplete. The originals stay in place with pointers to a new amendments section, so a later session can see what changed rather than reading a rewritten history. A1: D4 as written is unsound. The doc and overflow slabs are append-only, so replay repairs them, but B+tree node pages are mutated in place -- after a crash the file holds an arbitrary mix of written-back and not-written-back pages, and once D6.3 truncates the log the data file is the only copy below the watermark. A half-persisted tree is unrecoverable. So the checkpoint needs shadow paging: no page below the last watermark's allocation mark is ever stored into, and the watermark write is the atomic switch. Knock-on: node ids cannot be page numbers, because Node.parent/next/prev are back-pointers by id and copy-on-write would cascade; an in-RAM id->page table per index keeps every persisted id at its current width and gives COW one pointer to fix. A2: follows from A1 -- a page free list is a prerequisite, not the defense-in-depth D6.2 assumed, because COW abandons every page it touches in every epoch. The churn gate stays, retargeted at document garbage. A3: section 5 step 4's trap was misidentified. store_record already copies keys, so "entries must own their key bytes" is work that does not need doing. The real problem is that a leaf record has nowhere to put a slab offset; the resolution is to make the payload that offset, in every index, and delete Entry.id rather than re-own it. A4: making _id_ a unique index keys uniqueness on the canonical encode_key rather than serialize_value, so int32 1 / int64 1 / double 1.0 collide as they do in MongoDB -- a compatibility improvement, with a documented one-way migration hazard for a database that already holds two such documents. Also records that Engine.seq is never restored on open (harmless today, silent data loss once a watermark exists), and two bugs the new spec harness found. AGENTS.md carries the same rules into the operating guide: ground rules grow from 7 to 9, and the old rule 6 is corrected with a note saying why.
This commit is contained in:
198
AGENTS.md
Normal file
198
AGENTS.md
Normal file
@@ -0,0 +1,198 @@
|
||||
# AGENTS.md — operating guide for AI agents working in this repo
|
||||
|
||||
Read this first. It exists so any agent — in any session, without the
|
||||
original conversation's context — can work here safely and verifiably.
|
||||
|
||||
## What this is
|
||||
|
||||
MultiforaDB: a lightweight, embedded, MongoDB-compatible document database
|
||||
in Zig 0.16. It speaks the MongoDB wire protocol (OP_MSG, maxWireVersion 8)
|
||||
so real drivers (`mongosh`, Node, PyMongo) connect over TCP. Everything
|
||||
lives in a single-file LZ4-block log with an in-memory engine, B+tree
|
||||
indexes, and TTL/unique/sparse/compound index support.
|
||||
|
||||
**Forward plan**: the project's direction — a full-fledged embedded,
|
||||
tens-of-GB, maximally MongoDB-compatible database — is decided and written
|
||||
in **[PLAN.md](PLAN.md)**. The current milestone is **M0 (mmap + WAL
|
||||
storage foundation)**. Before starting any work, read PLAN.md; its decision
|
||||
record (D1-D9) and ground rules are binding.
|
||||
|
||||
## Read first, in order
|
||||
|
||||
1. **PLAN.md** — the plan: decisions, milestones M0-M9, gates, work
|
||||
breakdown, deferred designs. The next session always starts here.
|
||||
2. **README.md** — current state, features, benchmark table, known gaps,
|
||||
working-with-large-collections notes.
|
||||
3. **ROADMAP.md** — design write-ups of the completed performance work
|
||||
(B+tree, ordered `_id` index, compressed log, byte storage, decomposed
|
||||
locks). Contains the project's ground rules and verification recipes.
|
||||
|
||||
## Toolchain
|
||||
|
||||
- **Zig 0.16.0** — pinned. Do not assume a newer Zig is compatible.
|
||||
- **Node + the official `mongodb` driver** (in `tests/e2e/node_modules`)
|
||||
for e2e. `npm install` in `tests/e2e` if node_modules is missing.
|
||||
|
||||
## Build / test / run
|
||||
|
||||
```sh
|
||||
zig build # builds zig-out/bin/multiforadb (ReleaseFast by default)
|
||||
zig build test # unit tests, ReleaseFast
|
||||
zig build test -Doptimize=ReleaseSafe # unit tests with safety checks on
|
||||
zig-out/bin/multiforadb --port 27017 --db data.log
|
||||
```
|
||||
|
||||
**Critical traps:**
|
||||
|
||||
- **`zig build test` does NOT refresh the server binary.** After any change
|
||||
under `src/`, rebuild with `zig build` before restarting the server, or
|
||||
e2e will run against stale code and report failures that the source no
|
||||
longer explains. (This has bitten repeatedly; it is documented in
|
||||
`tests/e2e/README.md`.)
|
||||
- **Debug builds are 10-200x slower** than ReleaseFast on every path. Never
|
||||
benchmark (or time out waiting for) a Debug server. `zig build` defaults
|
||||
to ReleaseFast on purpose.
|
||||
|
||||
## E2E discipline (any change to src/)
|
||||
|
||||
```sh
|
||||
zig build # rebuild the server first!
|
||||
zig build test # ReleaseFast
|
||||
zig build test -Doptimize=ReleaseSafe
|
||||
```
|
||||
|
||||
Then, against a server on port 27020 (TTL suites need `--ttl-sweep-secs 1`):
|
||||
|
||||
```sh
|
||||
node tests/e2e/e2e.js # CRUD + operators + aggregate + errors
|
||||
node tests/e2e/e2e2.js concurrent
|
||||
node tests/e2e/e2e2.js crash-a # write 50 docs, kill -9 the server
|
||||
node tests/e2e/e2e2.js crash-b # restart, verify all 50 survived
|
||||
node tests/e2e/e2e3.js # secondary indexes
|
||||
node tests/e2e/e2e4.js # TTL indexes (server must run --ttl-sweep-secs 1)
|
||||
node tests/e2e/e2e6.js # self-contained full lifecycle (spawns its own server, incl. kill -9)
|
||||
```
|
||||
|
||||
Which suites to run for a given change:
|
||||
|
||||
- anything touching the write path or log format → the crash pair
|
||||
(e2e2 crash-a/b) and e2e6
|
||||
- anything touching indexes → e2e3.js and e2e4.js
|
||||
- everything → all of the above
|
||||
|
||||
`tests/e2e/README.md` has the full matrix, ports, and harness docs
|
||||
(`big.js` for multi-GB loads, `compare-run.sh`/`bench-run.sh` for the
|
||||
benchmark gate).
|
||||
|
||||
## Spec tests (the compatibility gate — PLAN D2)
|
||||
|
||||
```sh
|
||||
bash tests/spec/fetch.sh # pinned mongodb/specifications suites
|
||||
node tests/spec/run.js --scorecard # run + re-record tests/spec/scorecard.txt
|
||||
node tests/spec/run.js --file find.json --verbose
|
||||
```
|
||||
|
||||
`tests/spec/scorecard.txt` is committed and re-recorded every milestone (PLAN
|
||||
D9), so progress is verifiable across sessions. It is a *measuring* tool: a red
|
||||
baseline is the expected state at M0, `run.js` exits 0 as long as it ran, and
|
||||
nothing unimplemented is ever counted as a pass. `tests/spec/README.md` has the
|
||||
scope, the pins, and what is not yet asserted.
|
||||
|
||||
## Ground rules (binding — from ROADMAP.md and PLAN.md)
|
||||
|
||||
1. **Measure A/B on one harness. Do not trust the model.** Several
|
||||
predictions made during the existing work were wrong in both directions.
|
||||
The cheap A/B is: flip one line, rebuild, run, flip it back.
|
||||
2. **Mutation-check any test guarding an invariant.** Break the thing the
|
||||
test is supposed to catch and confirm it goes red before trusting it.
|
||||
3. **The index invariant is absolute** (`src/index.zig` header): an index
|
||||
only *generates candidates*; the full filter is re-applied afterwards.
|
||||
Over-approximating is slow. Under-approximating is a wrong answer.
|
||||
4. **The database must always open.** Replay never refuses to start over
|
||||
recoverable damage (e.g., a unique index finding duplicates in existing
|
||||
data warns and keeps going).
|
||||
5. **The checkpoint never describes a state ahead of the durable log
|
||||
tail** (M0 invariant): watermark advancement is the last step of the
|
||||
checkpoint protocol, after the data-file fsync.
|
||||
6. **No page below the stable mark is ever stored into** (M0, PLAN
|
||||
Amendment A1). The whole crash story is downstream of this one rule: a
|
||||
mutation that would dirty a page belonging to the last durable
|
||||
checkpoint copies it to a fresh page first. Enforced structurally by a
|
||||
copy-on-write page accessor, and mechanically by `mprotect`-ing the
|
||||
stable prefix read-only in ReleaseSafe/test builds — a missed COW then
|
||||
segfaults in the suite instead of corrupting a database silently.
|
||||
7. **A checkpoint never renumbers slab offsets** (M0, Amendment A3). Index
|
||||
leaves hold physical document offsets; only a full rebuild may move
|
||||
documents, and it rebuilds every index in the same pass.
|
||||
8. **Write-then-extend discipline for mmap**: ftruncate before touching
|
||||
new pages; never fault past the end of the mapped file (SIGBUS).
|
||||
9. **`Engine.seq` is restored on open** as `max(watermark.seq, max replayed
|
||||
record.seq)`, and a log rewrite never emits a record whose seq is ≤ the
|
||||
watermark. Today `seq` restarts at 0 on every open — harmless without a
|
||||
watermark, silent data loss with one.
|
||||
|
||||
Note for anyone reading an older revision: rule 6 previously read "index
|
||||
entries own their key bytes". That was a misdiagnosis — `store_record`
|
||||
already copies keys. See PLAN Amendment A3 for what the actual trap was.
|
||||
|
||||
## Code style
|
||||
|
||||
Binding reference: **`docs/TIGER_STYLE.md`** (TigerBeetle's TigerStyle,
|
||||
adopted verbatim). Read it before writing code. Its mechanical rules are
|
||||
enforced: `zig fmt` clean, 4-space indent, hard 100-column line limit, no
|
||||
`if` without braces unless it fits on one line, snake_case for functions/
|
||||
variables/files, units last in names (`latency_ms_max`), and always-on
|
||||
assertions (the repo's `assert.zig`, see below).
|
||||
|
||||
Project-specific rules and deliberate deviations from TigerStyle:
|
||||
|
||||
- Zig 0.16 idioms: `std.Io` threaded through everything, unmanaged
|
||||
containers, arena-threaded allocators where the code does.
|
||||
- House style: user-declared functions use `snake_case`; see existing
|
||||
`src/` for the pattern. Keep `src/lib.zig` as the library root and keep
|
||||
the server (`src/server.zig`) a thin front-end — the C API seam (PLAN
|
||||
D1) depends on that staying clean.
|
||||
- Public code invariants are asserted with the repo's own `assert.zig`
|
||||
(always active, including ReleaseFast) rather than `std.debug.assert`,
|
||||
so they do not silently vanish in optimized builds.
|
||||
- **Deviation (allocation):** TigerStyle's "statically allocated at
|
||||
startup, no allocation after init" is not yet this repo's architecture —
|
||||
the engine is arena-threaded. Do not introduce new *long-lived* dynamic
|
||||
allocations on hot paths; arenas scoped to a request are fine. Static
|
||||
allocation is a deferred design item (PLAN).
|
||||
- **Deviation (recursion):** TigerStyle bans recursion. `bson.zig`'s
|
||||
document parser is still recursive (nested docs/arrays); keep new code
|
||||
iterative and do not extend the recursion depth usage without a design
|
||||
note. Bounding parser depth is tracked work.
|
||||
- **Functions stay under 70 lines** (TigerStyle hard limit). A handful of
|
||||
pre-existing offenders remain (see the reflow/refactor backlog in
|
||||
ROADMAP notes); new code must not add to the count.
|
||||
|
||||
## Source map
|
||||
|
||||
## Source map
|
||||
|
||||
```
|
||||
src/bson.zig BSON parse/serialize, ObjectId, canonical comparison/key encoding
|
||||
src/wire.zig OP_MSG/OP_QUERY framing, message + reply builders
|
||||
src/commands.zig command dispatch (hello, CRUD, aggregate, admin, indexes)
|
||||
src/server.zig TCP accept loop, per-connection handlers, TTL sweep monitor
|
||||
src/db.zig engine: db → collection → _id → document maps, slab storage
|
||||
src/storage.zig append-only log: blocks, LZ4, XxHash3, replay, compaction
|
||||
src/query.zig filter matcher, regex engine, sort, projection
|
||||
src/index.zig B+tree indexes: entries, search, query planner
|
||||
src/update.zig update operators with dot-path navigation
|
||||
src/main.zig CLI: --port, --bind, --db, --ttl-sweep-secs, --compact-threshold
|
||||
```
|
||||
|
||||
## Working on a milestone
|
||||
|
||||
1. Re-read PLAN.md (and this file). Confirm which milestone is current.
|
||||
2. Per-milestone gates are in PLAN.md; the M0 gate is 6 items ending with
|
||||
benchmark parity (phase8 baseline) and the spec-test runner red
|
||||
baseline.
|
||||
3. Commit scorecard and benchmark results with each milestone (PLAN D9) so
|
||||
progress stays verifiable across sessions.
|
||||
4. Deferred designs (cursors, aggregation, transactions, change streams,
|
||||
C API) are deliberately *not* specified yet — grill the design with the
|
||||
human before implementing (PLAN section 6).
|
||||
Reference in New Issue
Block a user