e416ad179a
Records what `lsid` support is and what it deliberately is not, so M4 inherits the decisions rather than the questions. The part worth keeping is not the design but its corrections: three of the assumptions this stage was planned on turned out to be wrong when measured against a real mongod, and one of them -- that unknown fields inside `lsid` are tolerated -- would have shipped a divergence nothing in the test corpus could have caught. Also states that the scorecard did not move, 194/97/196 either side, which was the prediction rather than a surprise: the corpus has no session entities at all. What is new is that the prediction is now checkable -- after the event assertions landed, a command wrongly refused here would show up as a changed event stream instead of silently.
891 lines
51 KiB
Markdown
891 lines
51 KiB
Markdown
# PLAN — from MVP to a full-fledged embedded MongoDB-compatible database
|
||
|
||
Status: **design agreed** (decision record below). Next action: **implement
|
||
milestone 0** (mmap + WAL storage foundation). This document is the
|
||
single source of truth for the project's direction; every milestone starts
|
||
by re-reading it. It is intentionally self-contained so work can continue
|
||
across sessions/agents without this planning conversation's context.
|
||
|
||
The current engine is documented in [README.md](README.md); the completed
|
||
performance work (B+tree, ordered `_id` index, compressed log, byte
|
||
storage, decomposed locks) is written up in [ROADMAP.md](ROADMAP.md).
|
||
Both stay current; this file adds the *forward* plan.
|
||
|
||
---
|
||
|
||
## 1. Goal and constraints
|
||
|
||
Build on the existing MVP a **full-fledged embedded document database that
|
||
is maximally MongoDB-compatible**, holding **tens of GB with excellent
|
||
performance** on ordinary machines.
|
||
|
||
- **Embedded** means: a host application runs the engine in-process; real
|
||
MongoDB drivers (`mongosh`, Node driver, PyMongo, …) connect over TCP (or
|
||
a unix socket) to a server the host starts. The engine is a library
|
||
(`src/lib.zig` already is one); `src/server.zig` is a thin front-end. A
|
||
stable C API is a later layer — the architecture must keep that seam
|
||
clean (D1).
|
||
- **Maximally compatible** is measured objectively: the official MongoDB
|
||
JSON specification test suites are the semantic gate, layered on top of
|
||
the existing hand-written e2e suites, which cover what the spec tests do
|
||
not (crash durability, compaction, TTL, lifecycle) (D2).
|
||
- **Tens of GB** forces a disk-backed engine: the current all-in-RAM model
|
||
costs ~1.5-2x data size in RSS (5 GB → 6-7 GB RAM), which is out of scope
|
||
for the target. Storage must therefore move to disk while keeping the
|
||
engine's behavior identical (D3, D4).
|
||
- **Excellent performance** is guarded by a benchmark-parity gate: no
|
||
existing benchmark row may regress against the recorded phase8 baseline
|
||
(D7).
|
||
|
||
### Out of scope for "full-fledged v1" (explicit, reconsidered only later)
|
||
|
||
- Replica sets, sharding, Atlas Search, time-series collections.
|
||
- Text and geo indexes (2d / 2dsphere). Unique / sparse / TTL / compound /
|
||
hashed / partial are the index surface.
|
||
- Full ICU collation — only the `simple` locale and a hand-rolled subset.
|
||
- OP_COMPRESSED (deferred to the last milestone; drivers negotiate and
|
||
tolerate refusal).
|
||
- Multi-process access to one data directory (like MongoDB, single-process
|
||
only; no file locking).
|
||
- GridFS needs no engine work: it is client-side and works over the wire
|
||
protocol once cursors/aggregation are in place.
|
||
|
||
---
|
||
|
||
## 2. Decision record
|
||
|
||
### D1 — Embedding model: in-process server now, architected as a library + C API later
|
||
|
||
The engine stays a library; the server is a thin front-end (177 lines
|
||
today). A host app starts the engine and runs the accept loop on its own
|
||
thread (or a unix socket). Compatibility is exercised through real drivers.
|
||
A stable C API is deliberately a *later* layer, not a current requirement —
|
||
a bespoke API can never be "MongoDB-compatible" anyway; only the wire
|
||
protocol can. The engine's public surface must stay clean enough that the
|
||
C API can be bolted on without a redesign.
|
||
|
||
### D2 — Compatibility yardstick: official spec tests + own e2e (hybrid)
|
||
|
||
- **Gate for command semantics**: the official MongoDB JSON spec tests
|
||
(github.com/mongodb/specifications) — crud, aggregate, sessions,
|
||
transactions, change streams, gridfs, collation (subset). This turns
|
||
"maximally compatible" into a concrete list of test files.
|
||
- **Runner**: Node-based (the official `mongodb` driver is already a dev
|
||
dependency). Pin both a specific commit of `mongodb/specifications` and a
|
||
specific driver version (lockfile).
|
||
- **Scorecard**: a committed file — **`tests/spec/scorecard.txt`**, with the
|
||
runner and its pin in `tests/spec/` (see `tests/spec/README.md`) — holding
|
||
per-file pass/fail/skip counts and every non-passing case with its reason,
|
||
updated every milestone. Objective progress, mirroring how benchmark
|
||
results are already recorded.
|
||
Two properties it must keep, or the number flatters the engine: nothing
|
||
unimplemented is ever counted as a pass (it is a SKIP with a reason), and
|
||
the one assertion class the runner does not yet check — `expectEvents`,
|
||
i.e. command monitoring — is disclosed at the top of the file, so `pass`
|
||
reads as an upper bound until M1 wires events up.
|
||
- **Own e2e stays**: crash pair, kill -9, compaction, TTL, concurrent
|
||
clients, lifecycle (e2e.js … e2e6.js). The spec tests do not cover these.
|
||
|
||
### D3 — Target scale: tens of GB → disk-backed engine required
|
||
|
||
All-in-RAM is ruled out at the target scale. The spec tests do not require
|
||
bounded memory — **MongoDB itself is not memory-bounded** (WiredTiger
|
||
caches ~50% of RAM) — but the *embedded* product requirement does. This
|
||
decision drives D4 and the milestone order (D5).
|
||
|
||
### D4 — Storage architecture: mmap data file + existing log as WAL
|
||
|
||
*Amended 2026-08-03 (see "Amendment A1" below): the checkpoint requires
|
||
copy-on-write. The rest of this decision stands as written.*
|
||
|
||
The data file holds the engine's structures as mmap'd regions; the
|
||
existing append-only log remains the **source of truth** (WAL) with a
|
||
**lagging checkpoint**:
|
||
|
||
- Documents live in a slab; B+tree nodes live in a flat u32-addressed
|
||
arena with an overflow slab for long keys. u32 addresses are *file
|
||
offsets*: **on-disk format == in-memory format**, no serialization on
|
||
page-in. (The ROADMAP chose the flat node array and the slab precisely so
|
||
this checkpoint story works.)
|
||
- Caching and eviction are the OS page cache's job (LMDB-style). RSS =
|
||
working set, not data size.
|
||
- The log keeps its exact record framing, torn-tail handling, group commit
|
||
and fsync-before-ack. The checkpoint may lag: on open, map the data file
|
||
and replay log records with seq > watermark. `Engine.apply_record` is
|
||
already idempotent.
|
||
- **No page below the last watermark's allocation mark is ever stored
|
||
into.** A mutation that would dirty such a page copies it to a freshly
|
||
allocated page first, and the watermark write is the atomic switch to the
|
||
new set of pages (Amendment A1). This is the premise the crash-recovery
|
||
invariant below actually rests on.
|
||
- **The in-memory docs hashmap is dropped.** At tens of GB it costs ~64-100
|
||
bytes/doc (10+ GB of RAM at 100M docs). The ordered `_id_` B+tree becomes
|
||
the primary lookup; scans become index-ordered. Trap: the B+tree leaf has
|
||
nowhere to put a document's slab offset — see Amendment A3, which
|
||
supersedes the "entries must own their key bytes" framing.
|
||
- Crash-recovery invariant, to hold absolutely:
|
||
**the checkpoint (data file) never describes a state ahead of the
|
||
durable log tail** (D6).
|
||
|
||
### D5 — Sequencing: narrow mmap foundation first, compatibility after
|
||
|
||
Milestone 0 is the riskiest technical piece (it touches every data
|
||
structure) and everything later — cursor internals (offsets vs pointers),
|
||
transactions (undo/COW), change streams (oplog tagging) — is built on top
|
||
of it. Building compatibility on the in-memory engine first would mean
|
||
writing those features twice. Therefore: **M0 = narrow mmap+WAL
|
||
foundation**, scoped tightly (below), with the spec-test runner built in
|
||
parallel (it is independent); all compatibility features land on the final
|
||
substrate.
|
||
|
||
### D6 — Foundation internals (agreed details for M0)
|
||
|
||
1. **One data file + the existing log.** Regions inside the data file:
|
||
header/metadata, B+tree node arena (+ overflow), doc slab (8 MiB
|
||
segments, as today). One file keeps backup/UX simple; growth is
|
||
ftruncate + remap (see 4).
|
||
2. **Page allocation: a free list is a prerequisite** (amended — see
|
||
Amendment A2; this item previously said "start with abandon semantics,
|
||
a freelist is defense-in-depth"). Copy-on-write abandons every node page
|
||
it touches in an epoch, *every* epoch, so a write-heavy workload grows
|
||
the file by `epochs × touched_set` without bound. The free list releases
|
||
pages with a **two-generation delay**, which also keeps generation N-1's
|
||
image intact and usable as a recovery fallback.
|
||
The **churn gate stays**, retargeted at *document* garbage in the slab:
|
||
after a 10-20 GB load with an index and ~50% churn (delete+reinsert),
|
||
the data file must be ≤ ~1.3x the live data. Dead document bytes are
|
||
reclaimed only by a full rebuild (D6.6/§5), never by a checkpoint,
|
||
because index leaves hold physical slab offsets.
|
||
3. **Watermark: double-buffered header slot** (two 4 KiB slots,
|
||
generation + checksum; pick the newer valid one) so a torn header write
|
||
can never advance the watermark past the durable log tail. Checkpoint
|
||
protocol, given the COW premise of D4 (*no page below the last
|
||
watermark's allocation mark has been stored into*): snapshot collections
|
||
(existing compaction machinery) → msync + fsync data regions → write the
|
||
catalog into freshly allocated pages, fsync → write watermark → fsync →
|
||
truncate log → advance the stable mark and rotate the free lists. The
|
||
watermark equals the group-commit sealed seq; it **never** moves past
|
||
the durable log tail. The single load-bearing ordering edge is that the
|
||
watermark write happens strictly *after* the data-page msync/fsync: the
|
||
watermark is a promise that every page it describes is on disk.
|
||
4. **File growth = ftruncate + remap.** u32 offsets are file offsets, not
|
||
pointers: even if remap moves the base, structures are unaffected.
|
||
Reserve address space for tens of GB (64-bit, MAP_NORESERVE).
|
||
5. **Single-process only** (documented; no file locking).
|
||
6. **Backup story** (a feature milestone, but the layout must not block
|
||
it): a `backup` command (checkpoint → consistent snapshot → copy),
|
||
like `sqlite3_backup`. Raw file copies are *not* consistent while the
|
||
checkpoint lags.
|
||
|
||
### D7 — Milestone 0 acceptance gate
|
||
|
||
1. Unit tests green in ReleaseFast **and** ReleaseSafe (existing
|
||
discipline).
|
||
2. All existing e2e green unchanged: e2e.js, e2e2.js (concurrent +
|
||
crash pair), e2e3.js, e2e4.js, e2e6.js — the mmap engine is a drop-in
|
||
replacement; no semantic regression.
|
||
3. Large smoke: 20-30 GB load via big.js; RSS ≈ working set, not data
|
||
size; reopen in seconds (replay from checkpoint must beat today's full
|
||
replay); kill -9 mid-write + reopen safe.
|
||
4. Churn gate (D6.2).
|
||
5. **Benchmark parity**: no phase8 row regresses (compare-run.sh +
|
||
concurrent.js) — the guardrail for "excellent performance".
|
||
6. The spec-test runner exists and runs the crud + aggregate suites
|
||
(red is fine at M0; it is the harness), scorecard file committed.
|
||
|
||
### D8 — Deferral policy
|
||
|
||
Deep design of cursors, the aggregation expression engine, transaction
|
||
snapshot isolation, and change-stream resume tokens is **deferred to their
|
||
own milestones**, when the substrate (mmap engine) is real — designs on
|
||
paper now would be for the wrong data model. Each such milestone gets its
|
||
own design review before implementation.
|
||
|
||
### D9 — Working mode
|
||
|
||
This plan is the shared context. Milestones proceed one at a time; each
|
||
starts by re-reading this document; the scorecard and benchmark results are
|
||
committed per milestone so progress is verifiable by any agent in any
|
||
session.
|
||
|
||
---
|
||
|
||
## 2.1 Amendments
|
||
|
||
Decisions D1-D9 were agreed before the M0 implementation review. That review
|
||
found three of them wrong or incomplete. The originals are left in place
|
||
above with pointers here, so a future session can see what changed and why
|
||
rather than reading a rewritten history.
|
||
|
||
### Amendment A1 — the checkpoint requires copy-on-write (amends D4, D6.3)
|
||
|
||
D4 as written is unsound. The doc slab and the overflow slab are
|
||
append-only, so replaying the log over them repairs whatever the checkpoint
|
||
missed. **B+tree node pages are mutated in place.** After a crash the file
|
||
holds an arbitrary mix of written-back and not-written-back pages; a tree
|
||
with half its dirtied pages persisted is not a tree, and replaying `upsert`
|
||
records into it does not repair it. There is no fallback either, because
|
||
D6.3's protocol truncates the log — below the watermark the data file is the
|
||
*only* copy.
|
||
|
||
So the data file must be a *consistent* snapshot at the watermark, which
|
||
forces shadow paging: **copy-on-first-touch above a stable mark, with the
|
||
watermark write as the atomic root switch.** LMDB's scheme, minus MVCC, plus
|
||
this project's WAL.
|
||
|
||
Recovery is then `image_G + replay(seq > W_G)` at every crash point, and it
|
||
holds *only* because no page in `image_G` was ever stored into. Consequence
|
||
for the hot path: **zero msyncs between checkpoints** (dirty pages may sit
|
||
in the page cache indefinitely; the kernel may write any of them back at any
|
||
time, and both outcomes are fine), so the write path keeps exactly one
|
||
fsync — the WAL's, in `Engine.commit`. Exactly one msync per checkpoint,
|
||
ordered strictly before the watermark write.
|
||
|
||
A knock-on: **node ids cannot be page numbers.** `Node.parent`, `next`,
|
||
`prev` and `Slot.extra` are back-pointers *by id*, so COWing a page would
|
||
force its siblings and its whole subtree to COW as well. A small in-RAM
|
||
`id -> page` table per index, written wholesale at each checkpoint, keeps
|
||
every persisted id at its exact current width and meaning and gives COW a
|
||
single pointer to fix. It costs one dependent load per node access and
|
||
4 B/node (~5.6 MB at 100M docs) against the 64-100 B/doc hashmap being
|
||
deleted. Dropping the back-pointers for an LMDB-style cursor path stack is
|
||
the better long-term shape and hands M1 its cursor stack for free — it is
|
||
the right post-M0 A/B, not an M0 prerequisite.
|
||
|
||
Alternative considered and rejected: a dirty flag plus a full rebuild on
|
||
unclean shutdown (no COW, no free list, much less code). Rejected because
|
||
every `kill -9` would then cost a full log replay — minutes at 20-30 GB,
|
||
against D7.3's "reopen in seconds" — and the log could never be truncated,
|
||
since a full rebuild must always remain possible.
|
||
|
||
### Amendment A2 — the page free list is a prerequisite (amends D6.2)
|
||
|
||
Follows from A1; D6.2 is rewritten in place. The "abandon semantics are
|
||
fine, a freelist is defense-in-depth" analysis was about *document*
|
||
garbage, where it is correct. COW garbage is different in kind.
|
||
|
||
### Amendment A3 — the trap is the leaf payload, not key ownership (amends D4, §5 step 4)
|
||
|
||
"When the map goes away, entries must own their key bytes" describes work
|
||
that does not need doing: `Index.store_record` already copies both the key
|
||
and the payload into the node page or the overflow slab, and `Entry.key` is
|
||
already an owned dupe. The real problem is that a leaf record has **nowhere
|
||
to put a document's slab offset** — `Slot.extra` is the payload length for a
|
||
leaf and the child node id for an internal separator.
|
||
|
||
Resolution: a leaf record becomes `key ++ offset_le`, so `extra` is always
|
||
8 and every byte-accounting site (`fits`, `record_cost`, `slot_cost`,
|
||
`balanced_cut`, `repack_keep_prefix`) is unchanged. `Entry.id` is **deleted**
|
||
rather than re-owned: every entry one document contributes shares one
|
||
payload, so the offset belongs on the mutation call, which also makes it
|
||
impossible to confuse the old offset (unique-check self-exclusion, entry
|
||
removal) with the new one. Records get *smaller*, and `lookup_eq` /
|
||
`lookup_range` come to yield `u64` values that are immune to tree mutation
|
||
instead of byte slices that alias tree pages.
|
||
|
||
Do this for **secondary** indexes too, not just `_id_`: it deletes the
|
||
`_id`-to-offset hash lookup on every index-driven query outright instead of
|
||
replacing it with a B+tree descent, and it is free on the write path because
|
||
a replace already removes and reinserts every entry in every index. The
|
||
price is the invariant now recorded in §4: a checkpoint may never renumber
|
||
slab offsets.
|
||
|
||
### Amendment A4 — `_id` uniqueness becomes canonical
|
||
|
||
`_id` uniqueness is enforced today by the docs hashmap, keyed on
|
||
`bson.serialize_value`, under which int32 `1`, int64 `1` and double `1.0`
|
||
are three distinct documents. The `_id_` B+tree is keyed on
|
||
`bson.encode_key`, which *is* canonical, so those three collide — as they do
|
||
in real MongoDB. Making `_id_` a `unique` index is therefore a
|
||
**compatibility improvement**, and it is what the guard comment at the top
|
||
of `src/index.zig` anticipated when it fenced off the old map fast path.
|
||
|
||
Migration hazard, accepted deliberately: a database written by the current
|
||
code may legitimately hold two such documents. Agreed with the human rather
|
||
than assumed.
|
||
|
||
*Corrected while implementing:* the hazard does **not** bite when `_id_`
|
||
becomes unique, and the reason matters for sequencing. Replay does not
|
||
maintain `_id_` entries — `apply_record` evicts through the docs map, keyed on
|
||
`serialize_value`, and the tree is bulk-built afterwards by
|
||
`build_all_indexes`. That build calls `finish_bulk` with enforcement off,
|
||
which tolerates duplicate keys and already warns ("unique index '_id_' has
|
||
duplicate keys in existing data"), honouring the "database must always open"
|
||
rule. So such a database reopens with both documents intact and a loud
|
||
warning, and new colliding inserts are rejected from then on.
|
||
|
||
The document *loss* arrives only with the commit that drops the docs hashmap,
|
||
because eviction then goes through the canonical tree and the second document
|
||
takes the first one's place. That commit is where this needs handling — a
|
||
pre-flight scan for compare-equal `_id`s, refusing to drop the map silently
|
||
while any exist — not here.
|
||
|
||
---
|
||
|
||
## 3. Milestones and gates
|
||
|
||
| # | Milestone | Scope | Gate |
|
||
|---|---|---|---|
|
||
| M0 | **mmap + WAL foundation** | data file format, page/extent allocator, mmap slab + B+tree arena, copy-on-write + page free list (A1/A2), watermark/replay, checkpoint (= compaction repurposed), leaf payload → slab offset (A3), drop docs hashmap, churn measurement | D7 (6 items) |
|
||
| M1 | **Cursors + wire polish** | getMore / killCursors / batchSize; server-side cursor state with idle timeout; sessions plumbing (lsid accepted) as drivers send it; hello advertisement updates; **`moreToCome` on requests** (see the bug below); command-monitoring assertions in the spec runner | crud spec suite green; e2e green |
|
||
| M2 | **Aggregation expansion** | pipeline stages and expression engine (tiered scope defined at M2 design review) | aggregate spec suite green |
|
||
| M3 | **Update operators + index types** | $setOnInsert, $addToSet, $mul, $min/$max, $pop, $pullAll, $currentDate, pipeline updates; partial + hashed indexes | remaining crud coverage; e2e3/e2e4 green |
|
||
| M4 | **Sessions + transactions** | logical sessions, snapshot isolation on the mmap engine, write concern at commit | sessions + transactions spec suites green |
|
||
| M5 | **Change streams** | change feed + resume tokens (likely log-seq based), getMore integration | change-streams spec suite green |
|
||
| M6 | **Admin/ops commands** | dbStats, collStats, serverStatus, ping, buildInfo, listDatabases filters, dropDatabase durability (log it) | mongosh UX smoke; e2e green |
|
||
| M7 | **Auth** | SCRAM-SHA-1/256, user management commands | auth spec suite (subset) green |
|
||
| M8 | **Collation** | `simple` locale + hand-rolled subset (case-insensitive etc.) | collation spec suite (passing files) green |
|
||
| M9 | **OP_COMPRESSED** | snappy/zstd codecs, negotiation | driver handshake with compression on |
|
||
| — | **C API** (after M9 or when demanded) | stable C surface over the engine; the seam preserved since D1 | C smoke tests |
|
||
|
||
Cross-cutting in every milestone: **error-code parity** with MongoDB for
|
||
the commands touched (the spec tests assert codes), scorecard + benchmark
|
||
results committed.
|
||
|
||
### Bug found by the spec harness: `$sort` in an aggregate crashes the server
|
||
|
||
**Severity: remote, client-triggerable heap corruption. Present at `d4c9b04`.**
|
||
Fix this before anything else in M0.
|
||
|
||
In `commands.cmd_aggregate`'s `$sort` stage, the branch that materializes a
|
||
not-yet-materialized stream builds its list with the *reply arena* and then
|
||
hands it to `trees`, whose `defer trees.deinit(ctx.gpa)` frees it with the
|
||
**gpa**:
|
||
|
||
```zig
|
||
var all: std.ArrayListUnmanaged(*const bson.Document) = .empty;
|
||
for (offs.items) |off| try all.append(arena, try doc_tree(arena, coll, off));
|
||
trees.deinit(ctx.gpa);
|
||
trees = all; // arena-owned buffer, gpa-freed at scope exit
|
||
```
|
||
|
||
So the list buffer is freed by an allocator that never owned it. macOS malloc
|
||
catches it and aborts (SIGTRAP, exit 133) with no panic text, which is why the
|
||
symptom reads as "the connection closed":
|
||
|
||
```
|
||
faulting thread: mfm_free <- Allocator.rawFree
|
||
<- array_list.Aligned(*const bson.Document).deinit
|
||
<- commands.cmd_aggregate
|
||
```
|
||
|
||
Trigger: **any pipeline with `$sort` and no preceding `$group`** — e.g.
|
||
`aggregate([{$sort: {x: 1}}])`. With a `$group` first, the stream is already in
|
||
tree form, the branch is skipped, and nothing happens. Every existing e2e
|
||
aggregate case happens to sort *after* grouping, which is exactly why this
|
||
survived: `e2e.js` and `e2e6.js` both use `$match + $group + $sum + $sort`.
|
||
|
||
Fix: allocate the list with `ctx.gpa` (the `*const bson.Document` values may
|
||
stay in the arena — it outlives the command; it is only the ArrayList's own
|
||
buffer whose allocator has to match its `deinit`). Add an e2e case for a bare
|
||
`$sort` pipeline, and mutation-check it by restoring `arena` on the append.
|
||
|
||
### Bug found by the spec harness: a nameless command leaked the catalog lock
|
||
|
||
**Fixed. Severity: permanent denial of service, remotely triggerable by an
|
||
ordinary query.** This one blocked the M0 scorecard outright and cost three
|
||
invalid baselines.
|
||
|
||
`dispatch` resolved the namespace *after* taking the catalog lock and bailed
|
||
with `orelse return` when the collection name was missing. A plain return runs
|
||
neither the `errdefer` nor the explicit unlocks, so the lock was held — shared —
|
||
for the life of the process. `db.aggregate(...)` reaches it: that sends
|
||
`{aggregate: 1}`, whose value is not a string.
|
||
|
||
Worth understanding how it hid, because the shape recurs: a leaked *shared* lock
|
||
is invisible to readers. `ping` and `listDatabases` kept answering in
|
||
microseconds and an external prober saw `ok 15ms` straight through the hang, so
|
||
the server looked healthy. Only a write needing the catalog exclusive to create
|
||
a collection blocked — so the damage appeared one command later, on a different
|
||
connection, as a client-side timeout with nothing linking it to the cause.
|
||
|
||
What actually found it: the driver's own command log, showing an insert sitting
|
||
for exactly `socketTimeoutMS` against an idle engine.
|
||
|
||
Namespace resolution now happens before any lock is taken. Two lessons kept in
|
||
`tests/spec/README.md`: a responsive server does not exonerate the engine, and
|
||
probe with the operation that is stuck rather than with `ping`.
|
||
|
||
### Bug found by the spec harness: unacknowledged writes corrupt the connection
|
||
|
||
**Fixed** (in M0 rather than M1 as originally recorded — it is a correctness
|
||
bug, not a missing feature, and it is a handful of lines).
|
||
|
||
`wire.Message.flags` is parsed and stored but **never read**. A driver sending
|
||
an unacknowledged write (`writeConcern: {w: 0}`) sets `moreToCome` (bit 0x2)
|
||
on the *request* and does not wait for a reply; the server replies anyway, so
|
||
that reply sits unread in the socket and every later command on that
|
||
connection reads the wrong one. Reproduced end to end:
|
||
|
||
```
|
||
acknowledged insert: ok
|
||
unacknowledged insert returned: {"acknowledged":false,"insertedId":2}
|
||
next command on same connection FAILED: MongoUnexpectedServerResponseError:
|
||
BSON element "cursor" is missing
|
||
```
|
||
|
||
`countDocuments` read the stale `insert` reply. So `w: 0` — a normal
|
||
performance choice — breaks a connection on first use, and it is invisible to
|
||
the existing e2e suites because none of them use it.
|
||
|
||
Fixed by suppressing the reply when `flags & 0x2` is set on an OP_MSG request:
|
||
the command still runs. The e2e case pins `maxPoolSize` to 1, since with a
|
||
larger pool the driver may hand the next operation a different connection and
|
||
hide it.
|
||
|
||
### What the harness was worth
|
||
|
||
Its first honest run — after the two bugs above and two genuine leaks in the
|
||
runner itself — reports **131 pass, 161 fail, 195 skip** over 175 files, with
|
||
zero timeouts. Before the catalog-lock fix the same suite reported 45 passes:
|
||
that gap is the measure of what one leaked lock was hiding, and of why a
|
||
scorecard is only worth committing once it disagrees with nothing that passes in
|
||
isolation.
|
||
|
||
The remaining failures are real work, and they cluster usefully: update-operator
|
||
gaps (`bad update`, `update must be a document` — M3), unimplemented commands
|
||
(`distinct`, `$merge`, `$out` — M2/M3), and result-shape mismatches in
|
||
`bulkWrite`/`insertMany`. That list, not the total, is the milestone backlog.
|
||
|
||
---
|
||
|
||
## 4. Ground rules (inherited and new)
|
||
|
||
From ROADMAP.md, still binding:
|
||
|
||
- **Measure A/B on one harness. Do not trust the model.**
|
||
- **Mutation-check any test guarding an invariant** — break it and confirm
|
||
it goes red.
|
||
- **The index invariant is absolute**: an index only generates candidates;
|
||
the full filter is re-applied afterwards. Over-approximating is slow;
|
||
under-approximating is a wrong answer.
|
||
- **The database must always open**: replay never refuses to start over
|
||
recoverable damage.
|
||
|
||
New for the mmap era:
|
||
|
||
- **The checkpoint never describes a state ahead of the durable log
|
||
tail.** Watermark advancement is the last step of the checkpoint
|
||
protocol, after the data-file fsync.
|
||
- **No page below the stable mark is ever stored into.** Everything else in
|
||
the crash story is downstream of this one rule (Amendment A1). It is
|
||
enforced structurally by routing every writable page access through a
|
||
copy-on-write accessor, and mechanically by `mprotect`-ing the stable
|
||
prefix read-only in ReleaseSafe and test builds so a missed COW segfaults
|
||
in the suite instead of corrupting a database silently.
|
||
- **A checkpoint never renumbers slab offsets.** Index leaves hold physical
|
||
offsets; only a full rebuild may move documents, and it rebuilds every
|
||
index in the same pass.
|
||
- **Index entries carry a slab offset, not a document id** once the docs
|
||
map is dropped (Amendment A3).
|
||
- **Write-then-extend discipline** for mmap: ftruncate before touching new
|
||
pages; never fault past the end of the mapped file (SIGBUS protection).
|
||
- Replay idempotence of `apply_record` is a *tested invariant*, not an
|
||
assumption — it is what makes a lagging checkpoint safe.
|
||
- **`Engine.seq` is restored on open** as `max(watermark.seq, max replayed
|
||
record.seq)`. Today it unconditionally restarts at 0 (`db.zig`) and
|
||
`apply_record` ignores `record.seq` entirely; harmless without a
|
||
watermark, silent data loss with one.
|
||
- **A log rewrite never emits a record whose seq is ≤ the data file's
|
||
watermark**, or the next open discards it. Today's
|
||
`compact_snapshot_coll` stamps every re-emitted record with the live
|
||
`self.seq`; the rebuild that replaces it emits an empty log instead.
|
||
|
||
---
|
||
|
||
## 5. Milestone 0 — work breakdown (where the next session starts)
|
||
|
||
The original eight-step list is refined into the ordered commit sequence
|
||
below (same work, sequenced so that **`zig build test` in ReleaseFast *and*
|
||
ReleaseSafe plus the e2e matrix stay green at every commit**). Steps 2-5
|
||
exist to shrink step 11, which is otherwise the one unavoidably large
|
||
change.
|
||
|
||
The data file's own format is recorded in `src/pager.zig`'s header comment,
|
||
in the style of the log format at the top of `src/storage.zig`. Its shape:
|
||
an array of 4 KiB pages with a single tail-bump extent allocator — page 0
|
||
the file header, pages 1-2 the watermark double buffer, pages 3.. data. D6.1
|
||
called for a region table, but the region count is dynamic and unbounded
|
||
(one node arena per index, one slab per collection), and N contiguous
|
||
regions cannot all grow at the tail. With extents there is exactly one
|
||
growth path, therefore exactly one place where the write-then-extend
|
||
discipline lives, and `ls`/`du` stay honest for D6.6's backup story.
|
||
|
||
| # | Commit | Notes |
|
||
|---|---|---|
|
||
| 0 | amend this document | Amendments A1-A4 |
|
||
| 1 | spec-test runner + scorecard | original step 1, still fully independent |
|
||
| 2 | tighten `reserve_for`'s node bound | `n*(depth+2+n/8)+4` demands ~528 MiB of headroom for a 1000-entry multikey batch; free as ArrayList capacity, real file growth once the arena is file-backed |
|
||
| 3 | accessors for node pages and overflow bytes | pure refactor over the existing ArrayList; add the missing comptime asserts pinning `@sizeOf(Node) == 4096` and `@sizeOf(Slot)` |
|
||
| 4 | heap-allocate `Index`; move its scratch buffers out | kills the 5 KB struct memmove in `Collection.remove_index` that dangles a live `Plan.index` |
|
||
| 5 | `_id_` becomes `unique`; add an exact point lookup | Amendment A4, including the replay warning and the test-corpus updates |
|
||
| 6 | leaf payload becomes the slab offset | Amendment A3; the docs map stays as a shadow cross-check that asserts both agree |
|
||
| 7 | streaming candidates + a reverse leaf iterator | a materializing full scan is ~160 MB at the 20 GB gate; cursors are M1 but *streaming a scan* is required here |
|
||
| 8 | `src/pager.zig`: page allocator, mmap over a fixed reservation | engine-unused; unit tests only |
|
||
| 9 | watermark double buffer, catalog stream, free list | engine-unused |
|
||
| 10 | the doc slab lives in the data file | offsets become absolute; the reservation moves ahead of the log append. **First measurable point: A/B bulk-insert throughput and RSS here, before the rest depends on it** |
|
||
| 11 | node arena and overflow live in the data file | 11a overflow, 11b node pages + the id→page table |
|
||
| 12 | copy-on-write above the stable mark; two-generation free list | inert while the stable mark is 0, so testable in isolation first |
|
||
| 13 | checkpoint + watermark; open from the checkpoint; restore `Engine.seq` | log **not** yet truncated, so full replay stays a live safety net and both paths are exercised |
|
||
| 14 | truncate the log after a durable checkpoint | the safety net comes out in its own small commit |
|
||
| 15 | `compact` becomes a data-file rebuild | `compact_snapshot_coll` deleted; same tmp+rename+retry shape |
|
||
| 16 | drop the docs hashmap | original step 7 |
|
||
| 17 | gates | D7's six items; churn gate per amended D6.2 |
|
||
|
||
All seventeen commits are landed. Both harness fixes that were prerequisites
|
||
for the gate rather than the work are in: `bench-run.sh` no longer copies its
|
||
report over `bench-latest.txt` after a degraded run (it used to, so the
|
||
baseline being defended could be clobbered), and the four B+tree dev harnesses
|
||
(`spill.zig`, `spill2.zig`, `stress.zig`, `fuzz_split.zig`) now have a
|
||
`zig build fuzz` step — they were in no build step, so `zig build test` could
|
||
not notice an API break in the one place that fuzzes splits and >1 KB keys.
|
||
|
||
### The gate result
|
||
|
||
Measured numbers, and the command that reproduces each, are in
|
||
`tests/e2e/results/m0-gates.txt`. In short:
|
||
|
||
| D7 | gate | outcome |
|
||
|---|---|---|
|
||
| 1 | unit tests RF + RS | pass, 122/122 both |
|
||
| 2 | e2e matrix unchanged | pass, every suite |
|
||
| 3 | 20-30 GB smoke, RSS ≈ working set, fast reopen | pass at 21.5 GB |
|
||
| 4 | churn gate | measured, bounded, above the hoped-for 1.3x |
|
||
| 5 | benchmark parity | pass except bulk insert, -24% |
|
||
| 6 | spec runner + scorecard | pass, scorecard unchanged |
|
||
|
||
Two of those need reading rather than a tick.
|
||
|
||
**D7.4** hoped for ~1.3x of live data and settles at 1.65x (delete-heavy) to
|
||
2.47x (update-heavy), flat in both cases. Before the gate's own findings were
|
||
fixed it was 4.1x and *climbing linearly* — nothing was being reclaimed at
|
||
all. The gate's stated purpose (amendment A2) was to decide whether doc-level
|
||
free lists are needed after M0, and the answer is yes, in M1: a rebuild needs
|
||
a whole second copy of the live data before the first can be freed, so
|
||
rebuild-only reclamation cannot reach 1.3x however it is tuned. What M0 owed
|
||
was a bound, and there is one.
|
||
|
||
**D7.5** says "no phase8 row regresses". Bulk insert throughput regresses
|
||
24%, reproducibly (732 → 555 MB/s). That is risk 1 as written: document bytes
|
||
now reach the disk uncompressed in the data file on top of the LZ4 log, and
|
||
that writeback bandwidth is new. Every other row is inside this machine's
|
||
run-to-run spread, which two consecutive runs of the same binary showed to be
|
||
27-51% on the sub-10 ms rows — so the gate is met for the read and latency
|
||
rows and not for bulk load. `createIndex` improves 62%, also reproducibly,
|
||
from the same change.
|
||
|
||
### What the gates found
|
||
|
||
Five bugs, none of which any unit test or e2e suite had reached, and four of
|
||
which need either a long-running workload or a second concurrent writer to
|
||
appear at all:
|
||
|
||
1. The compaction trigger had been dead since commit 14 — it gated on
|
||
`log.data_bytes`, and truncating the log at every checkpoint zeroes that.
|
||
2. `page_mut_cow` asked `p >= stable_pages`, which is false for a recycled
|
||
page, so every write to one copied and freed it again.
|
||
3. First fit let copy-on-write's one-page requests shave the extents the doc
|
||
slab needs, so the free list drained every generation.
|
||
4. A rebuild's freed space took two unrelated checkpoints to become reusable,
|
||
so the next rebuild grew the file instead of reusing it.
|
||
5. `reserve_pages`' promise was a single counter on the pager, and two upserts
|
||
on different collections release it independently — so the first to finish
|
||
revoked the second's promise mid-write. This aborted the server at four
|
||
concurrent clients on the first concurrent benchmark since the data file
|
||
landed. Risk 3, whose mitigation (private pre-allocated runs) was listed in
|
||
this document and never built.
|
||
|
||
The lesson worth carrying into M1 is the shape of 1-4: every one is a
|
||
*reclamation* bug, invisible to any test that does not run long enough to
|
||
reach a steady state. The unit suite proved each mechanism works once. Only
|
||
the churn gate showed that none of them worked twice.
|
||
|
||
A compatibility gap also surfaced, out of M0's scope: `update.apply` rejected
|
||
any update document whose first key was not `$`, so `replaceOne`,
|
||
`findOneAndReplace` and `bulkWrite`'s `replaceOne` all failed with "bad
|
||
update". Fixed straight after the gate rather than deferred to a milestone,
|
||
since it was small and the scorecard put a number on it (below).
|
||
|
||
### After the gate, before M1: two CRUD corrections
|
||
|
||
Not milestone work — both came out of the gate run and were cheap enough that
|
||
deferring them would have cost more in explanation than in code.
|
||
|
||
**Replacement-style writes.** MongoDB decides operator-vs-replacement on the
|
||
update document's first field; a replacement replaces every field except `_id`,
|
||
which is immutable. Also refuses two options rather than ignoring them: `multi`
|
||
with a replacement, and `sort` on an update spec (a MongoDB 8.0 addition —
|
||
ignoring it would silently write a *different* document than the client asked
|
||
for). Took the scorecard from 131 pass / 161 fail to 161 / 131, sixteen files
|
||
improved, none regressed.
|
||
|
||
**`nModified` counted every write.** MongoDB counts a document as modified only
|
||
if the update altered it, and writes nothing when it did not. Decided in the
|
||
engine, where the document is already serialized and the check lands before the
|
||
log append, so a no-op now costs no log record, no fsync and no garbage. That in
|
||
turn exposed `_id` not being stored first: MongoDB moves it to the front
|
||
whatever order it arrives in, and the Node driver appends a generated `_id`, so
|
||
replacing a document with itself was a byte-level change. 163 pass / 129 fail.
|
||
|
||
The pattern worth noting for M1: both were found by *running* the suites, not by
|
||
reading them, and the second was only visible because the first stopped masking
|
||
it.
|
||
|
||
### Before the free list: eight bugs the M1 design work turned up
|
||
|
||
The doc-level free list multiplies traffic through exactly the reclamation
|
||
paths, so those paths were read closely before anything was built. Three of the
|
||
eight were found that way, by reading. Five were found by the tests written for
|
||
the other three — the same lesson M0's gate taught, arriving one milestone
|
||
early: a reclamation bug is invisible until something runs long enough, or
|
||
concurrently enough, to reach the state that exposes it.
|
||
|
||
1. **`write_freelist` counted its entries before allocating its own pages.**
|
||
The allocation goes through `take_free`, which `swapRemove`s an exact-fit
|
||
entry, so the loop wrote one entry fewer than the count it had already
|
||
committed to and the hash landed eight bytes short. `read_freelist` then
|
||
declared the list corrupt and dropped all of it. A one-page freelist stream
|
||
and a one-page hole are both the common case, so this fired at essentially
|
||
every reopen: the free list has been discarded on restart since it existed.
|
||
|
||
2. **The free lists were read and written with no lock.** `free_pages` appended
|
||
to `free_pending` while `publish` rotated the three lists under `alloc_lock`,
|
||
and `write_freelist` walked all three while a concurrent `free_pages` could
|
||
reallocate them. Found by the test written for (1).
|
||
|
||
3. **A checkpoint never gave back the generation it replaced.** The catalog
|
||
stream and the freelist stream are allocated fresh every publish and were
|
||
never freed, so an idle server grew forever.
|
||
|
||
4. **A checkpoint could publish a watermark above the durable log tail.** The
|
||
snapshot's `seq` check does not catch a writer that appended *before* the
|
||
walk started and has not committed yet; the window is as wide as an fsync.
|
||
This was an assertion, so the failure mode was a server abort under exactly
|
||
the load that makes checkpoints frequent — and without the assertion it is
|
||
the loss of an acknowledged write, since the truncation that follows a
|
||
checkpoint would discard the record. Now a retry: seal and re-snapshot.
|
||
|
||
5. **A document append could land in the published image.** `slab_reserve`
|
||
checks that the append cursor is writable; `publish` can clear the
|
||
unpublished set between that check and `slab_append`'s copy, because the log
|
||
append and its fsync sit in between. SIGBUS where `protect_stable` is
|
||
compiled in, a silent overwrite of durable data in ReleaseFast, where it is
|
||
not. Fixed with a pager-level append lock, held shared by appenders and
|
||
exclusively by `publish`; measured at no cost on the write path (8 clients ×
|
||
1500 inserts at `{w:1,j:true}`: 23779–24879 docs/s before, 23823–24452
|
||
after).
|
||
|
||
6. **`write_catalog` read `slab_extents` under only the shared catalog lock**,
|
||
while `slab_reserve` appended to that ArrayList under the collection's.
|
||
|
||
7. **The engine's counters were shared by writers holding no lock in common.**
|
||
`live_docs`, `dead_docs`, `live_bytes` and `dead_bytes` are updated by a
|
||
writer holding its own collection's lock and the catalog's shared — so two
|
||
writers on different collections lose each other's updates, and a reader had
|
||
no way to see the totals consistently with the per-collection figures they
|
||
are supposed to equal. Now `counter_lock`, a leaf, with a `Counters`
|
||
snapshot for the two readers that compare them. The only one of the eight
|
||
that is **not** mutation-checked: it aborted three of eight ReleaseSafe runs
|
||
once (8) made the checkpoint's consistency check reachable, and then would
|
||
not re-trigger in 34 further runs, on the reverted fix and on the pre-fix
|
||
revision alike. The rate depends on machine load. It stands on inspection,
|
||
and the concurrency test now asserts the identity once everything is quiet
|
||
rather than relying on catching the race in the act.
|
||
|
||
8. **The slab did not count what the appender skips.** `slab_used` only ever
|
||
grew by a document's length, so the gap left when a checkpoint pushes the
|
||
append cursor up to a system page, and the tail of an extent abandoned for a
|
||
document that no longer fits, were counted nowhere — real garbage,
|
||
invisible to the trigger that decides whether a rebuild is worth doing, and
|
||
part of the 1.65×/2.47× the churn gate measured.
|
||
|
||
Accounting is now an identity rather than four independent counters:
|
||
`dead_bytes` is the sum of `slab_used - live_bytes` over the collections that
|
||
exist. `read_catalog` recomputes it on open instead of trusting the watermark's
|
||
hint, `compact` recomputes it from what a rebuild leaves behind instead of
|
||
zeroing it, and `write_catalog` returns both sums for the checkpoint to assert
|
||
under the quiescence condition it already had. That assertion is what surfaced
|
||
(7) and what mutation-checks (5) and (8).
|
||
|
||
Two of the eight — the drop that charged a dropped collection's live bytes to
|
||
`dead_bytes`, and the counters — also changed what the watermark is for: its
|
||
`dead_bytes` field is now a hint for anything inspecting the header, not a
|
||
source of truth, because a collection dropped after the last checkpoint is gone
|
||
from the catalog and would still be charged for in the hint.
|
||
|
||
**Still open, deliberately.** `rebuild_collection` frees the pages it abandoned
|
||
while holding only the collection's lock, and a concurrent `checkpoint` may
|
||
already have snapshotted a catalog that claims them. The `seq` retry does not
|
||
see it, because a rebuild appends no log record. The fix is mutual exclusion
|
||
between `compact` and `checkpoint`; it is out of this scope because it wants
|
||
its own design pass, and because the free list must not add a second instance
|
||
of the same shape.
|
||
|
||
### The spec runner starts reading `expectEvents`
|
||
|
||
354 of the 487 cases declare `expectEvents` and the runner read none of them,
|
||
so a case could send the wrong command entirely and still be counted a pass as
|
||
long as the result came back right. The old `pass` column was an upper bound by
|
||
construction and said so; it is now an assertion that the engine answered
|
||
correctly **and** was asked the right question. **Scorecards recorded before
|
||
this are not comparable with ones recorded after.**
|
||
|
||
The totals moved 168/124/195 → 194/97/196 across the commits, but the path
|
||
matters more than the endpoints: turning the assertion on cost 34 passes, and
|
||
every one of them was a defect the result column could not see.
|
||
|
||
What it found, in the order it found them:
|
||
|
||
1. **The runner dropped `collectionOptions`.** Every collection entity was
|
||
built as `db.collection(name)`, so the 15 entities declaring
|
||
`writeConcern: {w: 0}` never got it — **every "unacknowledged write" case in
|
||
the corpus was running an acknowledged write.** They passed because the two
|
||
produce results a `$$unsetOrMatches` expectation accepts either way. Only
|
||
the command on the wire distinguished them, and nothing read the command.
|
||
2. **The wire version disagreed with the version string.** `buildInfo` said
|
||
4.4.0, the handshake said maxWireVersion 8, which is 4.2. A driver believes
|
||
the wire version: it refused *client-side* to send `hint` on an
|
||
unacknowledged delete or findAndModify, and withheld `comment` from
|
||
getMore, listCollections and listDatabases. 16 cases. The 8 was not
|
||
arbitrary — it was tied to keeping drivers off the streaming hello protocol
|
||
— but that turned out to rest entirely on omitting `topologyVersion`, which
|
||
is checked in the driver and is the whole mechanism. A test now asserts the
|
||
two numbers agree, since drifting apart silently was the actual defect.
|
||
3. **`$$unsetOrMatches` was changing root-ness.** The operator wraps a value,
|
||
it does not reposition it; the runner matched what stood behind it as a
|
||
nested document. 25 cases, all of them results the engine had right.
|
||
4. **An event's command is not the shape the driver sends.** A sort is held as
|
||
a JS `Map`, so `Object.keys` on it is empty and every expected key read as
|
||
missing. It hides well: EJSON prints a Map exactly like a document, so the
|
||
event dump reads as evidence the matcher is wrong about something else.
|
||
|
||
Two assertions are declined, both enumerated in the runner and in
|
||
`scorecard.txt`, and neither can hide anything the engine did:
|
||
|
||
- **`maxTimeMS`** — the harness's own doing. Every client carries CSOT
|
||
`timeoutMS`, which overwrites `maxTimeMS` with the remaining budget, so the
|
||
value on the wire is ours. Refused unconditionally rather than only when it
|
||
would fail, so it cannot become a pass by coincidence. One case, and dropping
|
||
`timeoutMS` instead would cost far more — it is what replaced the outer race
|
||
that once produced ~190 phantom timeout FAILs.
|
||
- **`cmap`/`sdam` event types, `ignoreExtraEvents`, `hasServiceId`,
|
||
`hasServerConnectionId`** — none occurs in this corpus; reported unsupported
|
||
where asserted rather than waived.
|
||
|
||
**Left failing on purpose: `bypassDocumentValidation: false`, 4 cases.**
|
||
mongodb@7.5.0 strips the field unless it is exactly `true` on the bulk and
|
||
findAndModify paths (`lib/bulk/common.js:292`,
|
||
`lib/operations/find_and_modify.js:19`) while sending it correctly for single
|
||
-document operations, so 4 sibling cases pass and 4 fail on a difference that
|
||
is entirely the driver's. The field is built client-side and never reaches the
|
||
engine. A refusal was written and thrown away: made unconditional it also
|
||
skipped the 4 that legitimately pass, and made conditional it would be a
|
||
skip-when-it-would-fail rule, which is the shape that turns a scorecard into
|
||
flattery. Four undeserved entries in the fail column is the cheaper error, and
|
||
this note is the correction. Revisit when the driver is bumped — which already
|
||
has to be its own commit with its own re-recorded scorecard.
|
||
|
||
---
|
||
|
||
## 6. Deferred designs (grill each at its milestone)
|
||
|
||
- **M1 cursors** — *settled and implemented.* The design lives in
|
||
`src/cursor.zig`'s module comment; the decisions it records, and how each
|
||
was reached:
|
||
- **Cursor ids** are `(nonce << 20) | slot`, always positive, never 0. The
|
||
nonce is not decoration: without it a recycled slot serves one client
|
||
another's documents, which is the worst failure this feature could have.
|
||
- **batchSize semantics** were *measured against mongod 8.3.7*, not
|
||
recalled, and three assumptions were wrong: a bare `getMore` does **not**
|
||
inherit the find's batchSize (4998 of 5000 documents come back), a
|
||
namespace mismatch is `Unauthorized` (13) rather than `CursorNotFound`,
|
||
and `CursorInUse` is 143 rather than the 12051 an earlier note claimed.
|
||
`internalQueryFindCommandBatchSize` is 101, `cursorTimeoutMillis`
|
||
600000, `clientCursorMonitorFrequencySecs` 4.
|
||
- **Never look ahead**: a batch that met its target leaves the cursor open
|
||
even when the source is in fact exhausted, so four documents at
|
||
`batchSize: 2` take three commands. The pinned suites assert that count.
|
||
- **Idle expiration** is a second monitor fiber, separate from the TTL one:
|
||
the cadences differ by an order of magnitude, and a TTL sweep failure
|
||
must not stop cursors being reclaimed. The registry is fixed-capacity and
|
||
evicts the least recently used cursor, whose client sees the same
|
||
`CursorNotFound` an idle timeout gives.
|
||
- **Against a lagging/compactable engine**, what survives depends on what
|
||
the cursor remembers, so the check is per-source: a repack changes no
|
||
key, so a streaming cursor resumes; slab offsets all move, so an offsets
|
||
cursor is killed with `QueryPlanKilled`; a snapshot needs no collection at
|
||
all. `Collection.layout_epoch` and `Index.epoch` are the tokens.
|
||
- **Resume** anchors on `(key, off)` plus a position hint gated on
|
||
`Index.epoch`, with an exact-order band walk bounded by
|
||
`resume_walk_max`. Two hazards found while implementing: a deleted anchor
|
||
must resume at its band position or the rest of an equal-key band is
|
||
silently dropped, and on a *unique* index a same-key entry can only be
|
||
the anchor rewritten — resuming at it returned updated documents twice,
|
||
caught by draining a collection being updated underneath.
|
||
|
||
Still open in M1: the doc-level free list. The eight reclamation bugs above
|
||
were cleared first, as preconditions for the free list rather than as work of
|
||
their own; command-monitoring (`expectEvents`) landed next, so that what
|
||
followed is measured by an instrument no longer known to overstate.
|
||
**A prerequisite the free list must honour**, recorded here while it is
|
||
still being designed: *an offset that was ever a record start must remain a
|
||
record start.* `doc_bytes` reads a `u32` length prefix in place, so an
|
||
offset landing mid-record after a re-split is a garbage-length read rather
|
||
than a wrong answer — and an offsets cursor holds exactly such offsets.
|
||
- **M1 sessions** — *settled and implemented.* `lsid` is parsed, validated and
|
||
deliberately acted on in no way; `txnNumber`, `startTransaction` and
|
||
`autocommit` are refused; `endSessions` validates the array it discards.
|
||
Every code and message was **measured against mongod 8.3.7** with a raw
|
||
OP_MSG probe, because the driver overwrites `lsid` with its own session and a
|
||
malformed one cannot be sent through it. Three measurements contradicted the
|
||
design they were checking:
|
||
- **Unknown fields inside `lsid` are rejected** (IDLUnknownField 40415). The
|
||
design said to tolerate them, reasoning that the server tolerates unknown
|
||
fields everywhere. It does not, here.
|
||
- **`uid` is accepted** — the hash of the credentials owning the session,
|
||
which a driver sends as soon as authentication is on. Rejecting it would
|
||
have broken every command in M7.
|
||
- **An unknown command with a malformed `lsid` answers CommandNotFound**, so
|
||
command lookup precedes session validation, which is where the check sits.
|
||
|
||
The refusals, so M4 does not reopen them:
|
||
- **No session registry.** A session here would own nothing: no transactions
|
||
to scope, no retryable writes (a driver disables them for a standalone),
|
||
and cursors that outlive their connection for reasons of their own. It
|
||
would be a mutex on the dispatch path guarding state nothing reads. M4's
|
||
transaction state machine gets to say what shape it needs.
|
||
- **`lsid` is not echoed.** Measured: mongod answers a well-formed one with
|
||
exactly `{ok: 1}`. A driver reads only `$clusterTime` and `operationTime`
|
||
back, and a standalone sends neither — correctly, since without
|
||
`operationTime` there is nothing for `afterClusterTime` to attach to and
|
||
causal consistency stays off.
|
||
- **`startSession` and `refreshSessions` are not implemented.** Both are real
|
||
mongod commands, but a driver calls neither — it generates session ids
|
||
locally — so CommandNotFound is the honest answer. Candidates for M4.
|
||
|
||
Five divergences from mongod remain, all deliberate. Three share one cause:
|
||
mongod keeps a per-command table of which commands accept `txnNumber` at all
|
||
and answers Location50889 or OperationNotSupportedInTransaction 263 for those
|
||
that do not, *before* reaching the standalone refusal. We have no such table
|
||
and give the standalone answer uniformly, so replies are identical for every
|
||
CRUD command — everything a driver would send these fields on — and differ
|
||
only on things like `ping`, where mongod is more specific rather than
|
||
differently right. The other two are `startSession`/`refreshSessions` above,
|
||
with `commitTransaction` alongside them, reachable only by a client whose
|
||
write this server has already refused.
|
||
|
||
**Effect on the scorecard: exactly zero, and that was the prediction.**
|
||
194/97/196 before and after. The corpus has no `session` entity, no operation
|
||
taking a `session` argument, and no `lsid` assertion. The value here is
|
||
protocol hygiene, not a number — and after Stage 1 a wrongly-refused command
|
||
would have shown up as a changed event stream rather than silently.
|
||
- **M2 aggregation**: stage/expression tiers, which spec-test files are
|
||
the gate, whether $lookup/$unwind/facet make the first cut.
|
||
- **M4 transactions**: snapshot isolation over mmap (COW vs undo), read
|
||
concern snapshot, conflict → TransientTransactionError semantics,
|
||
retryable-writes interplay.
|
||
- **M5 change streams**: resume-token design (log seq), live fan-out,
|
||
durability of the resume point.
|
||
- **C API**: exact surface (open/close, command exec, cursor iteration,
|
||
error reporting) — after the compat milestones, on the D1 seam.
|