Aleksey Shakhmatov
f2844e7894
Every reply came back in a single batch with `cursor.id = 0`, `getMore` was a
stub answering an empty `nextBatch` on the literal namespace `test.$cmd`, and
nothing read `batchSize`. That caps the useful collection size at what fits in
one 48 MiB message, which is the opposite of the tens-of-GB target and the
reason M0 made whole-index scans stream: the streaming candidate generator
existed with no consumer that could suspend.
## What a cursor is allowed to remember
A cursor holds no lock between requests, so everything it saves has to survive
arbitrary concurrent mutation. Nothing here is a pointer, and the two things
that look like stable addresses are not: `reset_tree` re-creates node ids 0 and
1 as different nodes, and `rebuild_collection` moves every document. Three
sources, chosen by query shape, each with a different memory contract:
- **stream** -- an index-ordered walk resumed from a `(key, off)` anchor plus a
`(leaf, slot)` hint. O(key) state, so this is what lets a cursor walk a
collection larger than memory. Survives a rebuild, because a repack changes
no key.
- **offsets** -- the matched slab offsets a narrowed plan already materialized,
8 bytes each. Killed by a rebuild with `QueryPlanKilled`, because those
offsets now name unrelated bytes.
- **buffered** -- canonical BSON copies, for a sort no index provides and for
aggregate/listing output. Depends on nothing, which is what lets a listing
hold a cursor over a `$cmd.*` namespace no collection backs.
`Collection.layout_epoch` and `Index.epoch` are the invalidation tokens, both
checked as error returns rather than assertions since a client reaches them by
keeping a cursor open across maintenance.
## Resume
`resume_forward`/`resume_reverse` are O(1) while the hint holds and fall back to
an exact-order band walk bounded by `resume_walk_max`. Without the hint, `seek`
lands at the *start* of an equal-key band, so `sort({status: 1})` over three
distinct values across 10M documents would cost ~5e10 comparisons to drain.
Two hazards found by draining a collection while writing to it, neither
predictable from reading the code:
- A deleted anchor must resume at its *band position*, or the rest of an
equal-key band is silently dropped -- most of the collection on a
low-cardinality index. Hence `band_index`.
- On a **unique** index a same-key entry can only be the anchor rewritten, so
resuming at it returned updated documents twice. Observed as duplicate `_id`s
while updating underneath a drain.
## Protocol
Measured against mongod 8.3.7 rather than recalled, which corrected three
assumptions: a bare `getMore` does *not* inherit the find's `batchSize` (4998 of
5000 documents come back), a namespace mismatch is `Unauthorized` (13) not
`CursorNotFound`, and `CursorInUse` is 143 not 12051.
`internalQueryFindCommandBatchSize` is 101, `cursorTimeoutMillis` 600000,
`clientCursorMonitorFrequencySecs` 4.
The rule everything follows is **never look ahead**: a batch that met its target
leaves the cursor open even when the source is in fact exhausted, so four
documents at `batchSize: 2` take three commands. `limit` acts as an EOF source,
which is what makes `batchSize == limit` close in one round trip. `skip` is
consumed once. `batchSize: 0` returns an empty batch with a live cursor.
Cursor ids are `(nonce << 20) | slot`, always positive. The nonce is not
decoration: without it a recycled slot serves one client another's documents.
Cursors are not connection-pinned, since the driver spec allows a `getMore` on
any connection to the same server; they end at exhaustion, `killCursors`, or the
idle sweep (a second monitor fiber, separate from the TTL one because the
cadences differ by an order of magnitude and a TTL failure must not stop
reclamation). The registry is fixed-capacity and evicts the least recently used
cursor, whose client sees the same 43 an idle timeout gives.
Fixed alongside, because cursors are what expose them:
- `listCollections` reported `"<db>."` with an *empty* collection part, which
makes the driver throw client-side -- so it would have broken the moment its
cursor stopped being id 0. Now `<db>.$cmd.listCollections`, as mongod uses.
- `count` ignored `skip` and `limit` entirely.
- `wire.end_message` now bounds a reply by the 48 MiB we advertise rather than
by `maxInt(u32)`; a reply past what we told the client to expect is not a large
reply, it is a desynchronized connection.
- Two `codeName` strings were wrong: 72 is `InvalidOptions` (MongoDB has no
`InvalidArgument`), and 40324 reports as `Location40324`.
## Verification
Unit 160/160 in ReleaseFast and ReleaseSafe; `tests/e2e/e2e7.js` adds 86 cursor
checks across five phases (batching/lifecycle/errors, streaming across churn,
aggregate+listings+count, expiry+capacity, restart) and is self-contained
because cursor behaviour is only observable with non-default flags. No
regressions: e2e 49, e2e3 16, e2e4 17, e2e2 2, e2e6 72. Spec 168 pass / 124
fail, +5 against the previous scorecard.
Mutation-checked, per the repo's second ground rule: `hint_slot + 1`, the
`band_index` off-by-one, both epoch bumps, the id nonce, the at-least-one-
document rule, and `stream_shape` returning null each turn the intended test
red. One claim was withdrawn rather than kept -- swapping `std.mem.order` for
`cmp_prefix` in the band walk changes nothing observable, so the comment now
says so instead of asserting a check that does not hold.
10 KiB
10 KiB
AGENTS.md — operating guide for AI agents working in this repo
Read this first. It exists so any agent — in any session, without the original conversation's context — can work here safely and verifiably.
What this is
MultiforaDB: a lightweight, embedded, MongoDB-compatible document database
in Zig 0.16. It speaks the MongoDB wire protocol (OP_MSG, maxWireVersion 8)
so real drivers (mongosh, Node, PyMongo) connect over TCP. Everything
lives in a single-file LZ4-block log with an in-memory engine, B+tree
indexes, and TTL/unique/sparse/compound index support.
Forward plan: the project's direction — a full-fledged embedded,
tens-of-GB, maximally MongoDB-compatible database — is decided and written
in PLAN.md. M0 (mmap + WAL storage foundation) has landed;
the current milestone is M1 (cursors + wire polish), whose cursor work is
done — see src/cursor.zig and tests/e2e/e2e7.js. Before starting any
work, read PLAN.md; its decision record (D1-D9) and ground rules are
binding.
Read first, in order
- PLAN.md — the plan: decisions, milestones M0-M9, gates, work breakdown, deferred designs. The next session always starts here.
- README.md — current state, features, benchmark table, known gaps, working-with-large-collections notes.
- ROADMAP.md — design write-ups of the completed performance work
(B+tree, ordered
_idindex, compressed log, byte storage, decomposed locks). Contains the project's ground rules and verification recipes.
Toolchain
- Zig 0.16.0 — pinned. Do not assume a newer Zig is compatible.
- Node + the official
mongodbdriver (intests/e2e/node_modules) for e2e.npm installintests/e2eif node_modules is missing.
Build / test / run
zig build # builds zig-out/bin/multiforadb (ReleaseFast by default)
zig build test # unit tests, ReleaseFast
zig build test -Doptimize=ReleaseSafe # unit tests with safety checks on
zig-out/bin/multiforadb --port 27017 --db data.log
Critical traps:
zig build testdoes NOT refresh the server binary. After any change undersrc/, rebuild withzig buildbefore restarting the server, or e2e will run against stale code and report failures that the source no longer explains. (This has bitten repeatedly; it is documented intests/e2e/README.md.)- Debug builds are 10-200x slower than ReleaseFast on every path. Never
benchmark (or time out waiting for) a Debug server.
zig builddefaults to ReleaseFast on purpose.
E2E discipline (any change to src/)
zig build # rebuild the server first!
zig build test # ReleaseFast
zig build test -Doptimize=ReleaseSafe
Then, against a server on port 27020 (TTL suites need --ttl-sweep-secs 1):
node tests/e2e/e2e.js # CRUD + operators + aggregate + errors
node tests/e2e/e2e2.js concurrent
node tests/e2e/e2e2.js crash-a # write 50 docs, kill -9 the server
node tests/e2e/e2e2.js crash-b # restart, verify all 50 survived
node tests/e2e/e2e3.js # secondary indexes
node tests/e2e/e2e4.js # TTL indexes (server must run --ttl-sweep-secs 1)
node tests/e2e/e2e6.js # self-contained full lifecycle (spawns its own server, incl. kill -9)
node tests/e2e/e2e7.js # self-contained cursors (spawns its own servers; needs no server running)
Which suites to run for a given change:
- anything touching the write path or log format → the crash pair (e2e2 crash-a/b) and e2e6
- anything touching indexes → e2e3.js and e2e4.js
- anything touching cursors, batching or the reply size → e2e7.js
- everything → all of the above
tests/e2e/README.md has the full matrix, ports, and harness docs
(big.js for multi-GB loads, compare-run.sh/bench-run.sh for the
benchmark gate).
Spec tests (the compatibility gate — PLAN D2)
bash tests/spec/fetch.sh # pinned mongodb/specifications suites
node tests/spec/run.js --scorecard # run + re-record tests/spec/scorecard.txt
node tests/spec/run.js --file find.json --verbose
tests/spec/scorecard.txt is committed and re-recorded every milestone (PLAN
D9), so progress is verifiable across sessions. It is a measuring tool: a red
baseline is the expected state at M0, run.js exits 0 as long as it ran, and
nothing unimplemented is ever counted as a pass. tests/spec/README.md has the
scope, the pins, and what is not yet asserted.
Ground rules (binding — from ROADMAP.md and PLAN.md)
- Measure A/B on one harness. Do not trust the model. Several predictions made during the existing work were wrong in both directions. The cheap A/B is: flip one line, rebuild, run, flip it back.
- Mutation-check any test guarding an invariant. Break the thing the test is supposed to catch and confirm it goes red before trusting it.
- The index invariant is absolute (
src/index.zigheader): an index only generates candidates; the full filter is re-applied afterwards. Over-approximating is slow. Under-approximating is a wrong answer. - The database must always open. Replay never refuses to start over recoverable damage (e.g., a unique index finding duplicates in existing data warns and keeps going).
- The checkpoint never describes a state ahead of the durable log tail (M0 invariant): watermark advancement is the last step of the checkpoint protocol, after the data-file fsync.
- No page below the stable mark is ever stored into (M0, PLAN
Amendment A1). The whole crash story is downstream of this one rule: a
mutation that would dirty a page belonging to the last durable
checkpoint copies it to a fresh page first. Enforced structurally by a
copy-on-write page accessor, and mechanically by
mprotect-ing the stable prefix read-only in ReleaseSafe/test builds — a missed COW then segfaults in the suite instead of corrupting a database silently. - A checkpoint never renumbers slab offsets (M0, Amendment A3). Index leaves hold physical document offsets; only a full rebuild may move documents, and it rebuilds every index in the same pass.
- Write-then-extend discipline for mmap: ftruncate before touching new pages; never fault past the end of the mapped file (SIGBUS).
Engine.seqis restored on open asmax(watermark.seq, max replayed record.seq), and a log rewrite never emits a record whose seq is ≤ the watermark. Todayseqrestarts at 0 on every open — harmless without a watermark, silent data loss with one.
Note for anyone reading an older revision: rule 6 previously read "index
entries own their key bytes". That was a misdiagnosis — store_record
already copies keys. See PLAN Amendment A3 for what the actual trap was.
Code style
Binding reference: docs/TIGER_STYLE.md (TigerBeetle's TigerStyle,
adopted verbatim). Read it before writing code. Its mechanical rules are
enforced: zig fmt clean, 4-space indent, hard 100-column line limit, no
if without braces unless it fits on one line, snake_case for functions/
variables/files, units last in names (latency_ms_max), and always-on
assertions (the repo's assert.zig, see below).
Project-specific rules and deliberate deviations from TigerStyle:
- Zig 0.16 idioms:
std.Iothreaded through everything, unmanaged containers, arena-threaded allocators where the code does. - House style: user-declared functions use
snake_case; see existingsrc/for the pattern. Keepsrc/lib.zigas the library root and keep the server (src/server.zig) a thin front-end — the C API seam (PLAN D1) depends on that staying clean. - Public code invariants are asserted with the repo's own
assert.zig(always active, including ReleaseFast) rather thanstd.debug.assert, so they do not silently vanish in optimized builds. - Deviation (allocation): TigerStyle's "statically allocated at startup, no allocation after init" is not yet this repo's architecture — the engine is arena-threaded. Do not introduce new long-lived dynamic allocations on hot paths; arenas scoped to a request are fine. Static allocation is a deferred design item (PLAN).
- Deviation (recursion): TigerStyle bans recursion.
bson.zig's document parser is still recursive (nested docs/arrays); keep new code iterative and do not extend the recursion depth usage without a design note. Bounding parser depth is tracked work. - Functions stay under 70 lines (TigerStyle hard limit). A handful of pre-existing offenders remain (see the reflow/refactor backlog in ROADMAP notes); new code must not add to the count.
Source map
Source map
src/bson.zig BSON parse/serialize, ObjectId, canonical comparison/key encoding
src/wire.zig OP_MSG/OP_QUERY framing, message + reply builders
src/commands.zig command dispatch (hello, CRUD, aggregate, admin, indexes)
src/server.zig TCP accept loop, per-connection handlers, TTL sweep monitor
src/db.zig engine: db → collection → _id → document maps, slab storage
src/storage.zig append-only log: blocks, LZ4, XxHash3, replay, compaction
src/query.zig filter matcher, regex engine, sort, projection
src/index.zig B+tree indexes: entries, search, query planner, scan resume
src/cursor.zig server-side cursor state: registry, batch policy, expiry
src/update.zig update operators with dot-path navigation
src/main.zig CLI: --port, --bind, --db, --ttl-sweep-secs, --compact-threshold
Working on a milestone
- Re-read PLAN.md (and this file). Confirm which milestone is current.
- Per-milestone gates are in PLAN.md; the M0 gate is 6 items ending with benchmark parity (phase8 baseline) and the spec-test runner red baseline.
- Commit scorecard and benchmark results with each milestone (PLAN D9) so progress stays verifiable across sessions.
- Deferred designs (aggregation, transactions, change streams, C API) are
deliberately not specified yet — grill the design with the human before
implementing (PLAN section 6). Cursors are no longer among them: the
design was settled and implemented in M1, and
src/cursor.zig's module comment is where it is written down.