The gate results file said the replacement-style-update gap was found and not
fixed, and PLAN said it was left alone. Both were true when written and are not
now, so a reader would take the M0 scorecard for the current one. The M0 figures
stay as measured -- they are the gate result -- with a pointer to
tests/spec/scorecard.txt, which always holds the current number.
PLAN D7's six items, with the numbers and the command that reproduces each in
tests/e2e/results/m0-gates.txt. Unit tests green in both optimize modes, the
whole e2e matrix green, the spec scorecard byte-identical at 131/161/195, and
the large smoke run at the scale D7.3 asked for:
21.47 GB collection (1,310,720 x 16 KiB)
data file 21.75 GB (+1.3% over the documents)
log after the load 2.5 MB (checkpoints reclaim it)
kill -9 then reopen 0.5 s (0.5 s at 4 GB too -- flat)
RSS after reopen 237 MB (1.1% of the data)
count after restart 1,310,720 last document byte-intact
acked writes after kill 200/200
That is the milestone's claim, measured: an open costs the working set rather
than the size of the database. Before M0 the same measurement was 523 MB
resident for a 512 MB database, because recovering each document's `_id` meant
reading every document at open.
Two gates need reading rather than a tick, and m0-gates.txt says so where a
reader would otherwise take a tick for granted.
The churn gate settles at 1.65x live data (delete-heavy) to 2.47x
(update-heavy), flat, above the ~1.3x amendment A2 hoped for. Rebuild-only
reclamation cannot reach that: it needs a whole second copy of the live data
before the first can be freed. The gate existed to decide whether doc-level
free lists are needed after M0, and that is the answer.
Benchmark parity holds for every read and latency row inside the run-to-run
spread, and bulk insert regresses 24% (732 -> 555 MB/s), reproducibly across
three runs. Risk 1 as written: document bytes now reach the disk uncompressed
on top of the LZ4 log. createIndex improves 62% from the same change.
Three measurement bugs fixed while running the gates, because each would have
put a false number in the README:
- `compare-run.sh` measured "db on disk" as `du` of the log alone against
`du` of mongod's whole dbpath. It reported 20 MB for a 1 GB collection --
the documents had moved to <db>.data. Honest figure, measured: 914 MB of
allocated blocks against mongod's compressed 85 MB.
- `big.js` counted "compaction events" as "the log shrank", which is a
*checkpoint* now. It claimed 12 compaction rewrites during a pure insert
load, which has no garbage to compact.
- `big.js` labelled peak RSS "in-memory engine: docs live in RAM" and its
summary said the collection was held "fully in RAM". Both were true of the
engine this milestone replaced.
README: the storage section described an all-in-RAM engine; the comparison
table mixed one old run's body with three new rows; and `findOne({_id})` was
documented as a full scan for integer ids, which the ordered `_id_` index made
false (2 ms against 55 s for a scan of the same 21.5 GB collection). The table
is now best-of-three for both servers, with the measured variance stated, since
two runs of the same binary moved the sub-10 ms rows by 27-51%.
PLAN D2 makes the official specification suites the gate for command semantics;
D7.6 asks for the harness to exist at M0 with a recorded baseline. This is that
harness, pinned on both sides -- mongodb/specifications @ 615e0f9 and
mongodb@7.5.0 -- because a scorecard is only comparable across milestones if a
delta cannot be an upstream test change.
It implements the unified format's Evaluating Matches algorithm as written,
including the two rules that decide whether a pass is earned: extra keys are
tolerated only in a root document, and numeric types compare flexibly. Anything
unimplemented is a SKIP with a reason, never a pass, and the one assertion class
not yet checked -- expectEvents, i.e. command monitoring -- is disclosed at the
top of the scorecard so `pass` reads as an upper bound.
First honest run: 131 pass, 161 fail, 195 skip over 175 files, zero timeouts.
Getting there took four attempts, and the failures are documented in the README
because each would have shipped a scorecard claiming a compatibility gap that
did not exist. Two were genuine leaks in this runner (clients left open when a
case timed out; clients registered for cleanup only after `await connect()`,
plus abandoned cases still creating more). The third I misdiagnosed as machine
load. The fourth attempt found the real cause: a leaked catalog lock in the
engine, fixed separately, which alone accounts for the jump from 45 passes to
131.
So the runner carries its own guards: per-operation CSOT timeouts so work is
never abandoned, an active-handle census per file, an end-of-run tripwire for
stray timers, a hard stop if the server dies rather than emitting hundreds of
misleading ECONNREFUSED failures, and --skip/--limit for bisecting a run whose
failures depend on position. The README states the rule plainly -- a long
unbroken tail of timeouts is a harness bug until proven otherwise -- and the two
commands that settle it.
Also fixes bench-run.sh, which copied its report over bench-latest.txt
unconditionally, including after a run that only warned -- so a degraded run
could silently replace the baseline that PLAN D7.5 makes a milestone gate.
_id uniqueness was a `coll.docs.contains` probe. The docs hashmap is going
away (PLAN A3), so it has to move to the _id_ tree -- and the tree answers
better, because it is keyed on bson.encode_key, which is canonical where
serialize_value is not. int32 1, int64 1 and double 1.0 are now one _id, as
they are in MongoDB (A4).
_id_ is built and checked first, so a write violating both it and a unique
secondary reports _id_, which is what MongoDB reports. It returns
error.DuplicateKey with `dup_index` left null, which is exactly what
commands.zig's E11000 rendering already treats as "the _id_ index", so the
wire-visible message is unchanged and that file needed no edit.
check_unique's exclude-self became optional and is null on an insert. That was
a latent bug of its own: a replace must ignore its own existing entries, but an
insert has none, and passing the document's id there hides a collision whose
entry carries that same id -- precisely the case _id_ exists to catch. Only
_id_ could reach it, since a secondary collision is between different
documents.
Two corrections found while doing this, both worth reading:
PLAN A4 claimed a database already holding {_id: int32 1} and {_id: int64 1}
loses one on reopen. It does not. Replay evicts through the docs map, keyed on
serialize_value, so both survive; the tree is bulk-built afterwards with
enforcement off, which tolerates duplicate keys and warns. The loss arrives
only with the commit that drops the map, and that is where it needs a
pre-flight scan. Amended.
dispatch_insert asserted only `ok: 1`, but a rejected document comes back as a
writeError alongside it -- so the mixed-type corpus silently shrank from ten
documents to nine when _id_ became unique, and every test over it still passed.
The helper now rejects writeErrors and asserts the inserted count; it caught
the shrink immediately. The corpus keeps an int64 _id on a distinct value, and
the collision it used to stand in for is asserted directly.
Also adds Index.lookup_exact, which the commands that currently probe the docs
map will need. Exact byte equality rather than cmp_prefix, because {a: 1}'s
encoding is a proper prefix of {a: 1, b: 2}'s and a prefix match would claim a
document is present when it is not.
Mutation-checked, all three red: unique=false on id_index; exclude=id_key on
insert; eql -> cmp_prefix in lookup_exact.
The M0 implementation review found three of D1-D9 wrong or incomplete. The
originals stay in place with pointers to a new amendments section, so a later
session can see what changed rather than reading a rewritten history.
A1: D4 as written is unsound. The doc and overflow slabs are append-only, so
replay repairs them, but B+tree node pages are mutated in place -- after a
crash the file holds an arbitrary mix of written-back and not-written-back
pages, and once D6.3 truncates the log the data file is the only copy below
the watermark. A half-persisted tree is unrecoverable. So the checkpoint needs
shadow paging: no page below the last watermark's allocation mark is ever
stored into, and the watermark write is the atomic switch. Knock-on: node ids
cannot be page numbers, because Node.parent/next/prev are back-pointers by id
and copy-on-write would cascade; an in-RAM id->page table per index keeps every
persisted id at its current width and gives COW one pointer to fix.
A2: follows from A1 -- a page free list is a prerequisite, not the
defense-in-depth D6.2 assumed, because COW abandons every page it touches in
every epoch. The churn gate stays, retargeted at document garbage.
A3: section 5 step 4's trap was misidentified. store_record already copies
keys, so "entries must own their key bytes" is work that does not need doing.
The real problem is that a leaf record has nowhere to put a slab offset; the
resolution is to make the payload that offset, in every index, and delete
Entry.id rather than re-own it.
A4: making _id_ a unique index keys uniqueness on the canonical encode_key
rather than serialize_value, so int32 1 / int64 1 / double 1.0 collide as they
do in MongoDB -- a compatibility improvement, with a documented one-way
migration hazard for a database that already holds two such documents.
Also records that Engine.seq is never restored on open (harmless today, silent
data loss once a watermark exists), and two bugs the new spec harness found.
AGENTS.md carries the same rules into the operating guide: ground rules grow
from 7 to 9, and the old rule 6 is corrected with a note saying why.
