fc611a2c6492b7f30c486e4d5e6dbe5032d54def
49 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
A.Shakhmatov
|
fc611a2c64 |
commands: $project computes, renames and narrows
The last three cases of the corpus, which is now 70 pass / 0 fail -- every
answer byte-identical to mongod 8.3.7 across the accumulators, the expressions
and the document stages.
The fix turned out to need nothing from `query.project`, which `find` shares
and which I had expected to have to rewrite. A nested spec *is* a dotted path:
`{n: {x: 1}}` and `{"n.x": 1}` are the same projection, and dotted paths are
something the existing projection already narrows correctly. So `$project` is
flattened into inclusion/exclusion flags plus a list of computed fields, and
both halves reuse machinery that was already there -- `query.project` for the
flags, `set_path` from the document stages for the computed fields. A bare path
(`{value: "$a"}`) is a rename, which is a computed field like any other.
Both shapes used to read as *falsy*, which flipped the whole projection into
its exclusion branch and returned the entire document minus the field. That was
recorded during M2 as broken rather than unimplemented; this is the fix it was
waiting for.
One case `query.project` genuinely cannot express, so it is built directly: a
projection that only computes keeps `_id` and nothing else, and with no non-`_id`
flag that function reads the spec as an exclusion and returns everything. It
cost two failures and a `id_only` flag to find, which is what a recorded corpus
is for -- the answer is obvious once seen and not before.
`$project` now goes through the same `Rewrite` path as `$addFields`, `$unset`,
`$replaceRoot` and `$unwind`, so its own branch is gone. Its refusal shrank to
the one shape mongod also refuses, mixing inclusion with exclusion, judged on
the flattened flags so a nested spec is treated like a dotted one.
191/191 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz, e2e 49, e2e2
concurrent, e2e3 16, e2e4 17, e2e6 72, e2e7 86, crud corpus unchanged at
201/90/196.
|
||
|
A.Shakhmatov
|
1a5386ff00 |
commands: the stages that rewrite a document
`$addFields`, `$set`, `$unset`, `$replaceRoot` and `$unwind`. The corpus goes
0 pass / 24 fail to 21 / 3, and the three left are `$project`'s computed
fields, renames and nested inclusions, which want the `query.project` that
`find` shares and are their own change.
**The design review was wrong about what this needed, and the corpus is what
settled it.** Tier 2 was scoped as a per-stage iterator on the grounds that
`$unwind` is 1->N and "there is no way to express that in a window over the
input". That was true of the window as it stood, and stopped being true the
moment `$project` was made to rebuild the stream instead of moving bounds over
it -- a stage that rebuilds can emit as many documents as it likes, or none. So
all five share one shape: read the window, build a new list, replace the
stream. No iterator, no rewrite.
What the recording settled, and what a hand-written test would have got wrong:
- `$addFields` whose expression resolves to nothing leaves the field out
entirely rather than setting it to null -- so `set_path` is only reached
when there is a value, and `eval_expr`'s absent/null distinction earns its
keep a second time.
- `$addFields: {"n.z": 1}` sets the nested path and keeps its siblings, and
an existing field is replaced *where it stands*, which is what makes the
stage "add or overwrite" rather than "append".
- `$unwind` drops a document whose field is missing or an empty array, keeps
one whose field is not an array *whole*, and numbers `includeArrayIndex`
from zero. Three separate behaviours where one guess would have covered
them all wrongly.
- `$replaceRoot` of a missing path and of a non-document are the same error,
40228.
`ReplaceRootNotDocument` joins `EvalError` rather than being reported at the
stage: it is a failure only a document can produce, which is the line that set
already draws.
191/191 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz, e2e 49, e2e3 16,
e2e4 17, e2e6 72, e2e7 86, crud corpus unchanged at 201/90/196.
|
||
|
A.Shakhmatov
|
88ac010c3c |
tests/spec: record Tier 2's spec -- the stages that rewrite a document
24 cases for `$addFields`/`$set`, `$unset`, `$replaceRoot`, `$unwind` and
`$project`'s computed fields, recorded from mongod 8.3.7 before any of them is
written. The file starts at 0 pass / 24 fail, which is the honest number for
five stages this server does not have.
Eight things the recording settled, none of them guessable from the manual:
$addFields whose expression is missing the field is not added at all
$addFields: {"n.z": 1} sets the nested path, keeps siblings
$replaceRoot of missing or non-document error 40228
$unwind of an empty array or missing the document is dropped
$unwind of a non-array the document is kept whole
$unwind path without a $ error 28818
includeArrayIndex 0-based
$project: {n: {x: 1}} narrows it; a document without
`n` keeps only `_id`
That last one is the `query.project` gap recorded during M2 as broken rather
than unimplemented -- a nested inclusion currently reads as falsy and returns
the whole document minus the field. It now has a measured expectation to be
fixed against, in the corpus, rather than a note in a commit message.
Worth stating before the implementation: the design review expected Tier 2 to
need a per-stage iterator, on the grounds that `$unwind` is 1->N and "there is
no way to express that in a window over the input". That was true of the window
as it stood, and stopped being true when `$project` was made to rebuild the
stream -- a stage that rebuilds can emit any number of documents it likes. So
Tier 2 looks like five stages rather than a rewrite. The corpus is what will
say whether that holds.
|
||
|
A.Shakhmatov
|
37cfa863ee |
commands: the aggregation expression evaluator
The whole corpus is green: 46 pass, 0 fail, byte-identical to mongod 8.3.7 on
every case including all 27 expressions and the compound `_id` that was the
last accumulator failure.
Expressions are *compiled once per pipeline and evaluated per document*, and
that split is the point rather than an optimisation: it keeps the property M2's
refusals bought, which is that a pipeline that cannot be answered is refused
before a single document is read instead of half way through with part of the
work already reported. `Expr` is the compiled tree, `compile_expr` reports,
`eval_expr` cannot.
Nineteen operators: `$literal`, the five arithmetic ones, seven comparisons,
`$and`/`$or`/`$not`, `$cond` in both its forms, `$ifNull` and `$switch`. Plus
the two shapes that are not operators at all -- a compound document, which is
what a `$group` `_id` usually is, and an array.
Everything the corpus recorded, and none of it guessable:
- absent and a present null are *different* internally, because `$ifNull`
treats them alike and `$push` does not. Hence `?bson.Value` throughout,
where the obvious shortcut is to fold absent into `.null` at the boundary
and lose the distinction for good.
- arithmetic over absent or null is `null` -- not an error, not zero -- and
over a string is an error, 7157723.
- `$divide` by zero is 4848401, `$switch` with no branch and no default is
40069, and both are failures only a document can produce, so `EvalError`
exists and `report_eval_error` maps it.
- two operators in one expression document is 15983 and *not* `$group`'s
40238: mongod distinguishes an expression from an accumulator there.
- truthiness is MongoDB's, so `-5` is true and `0.0` is false.
- `$mod` follows the dividend's sign, so -5 mod 4 is -1.
- `$not` takes a bare argument as readily as a one-element array.
`compile_expr` and `compile_operator` call each other, so their error set is
written out rather than inferred -- Zig cannot infer a cycle, and the failure
mode is a "dependency loop" message that says nothing about expressions.
Three cases left the Tier 0 refusal test, because a compound `_id`, `$literal`
and a `$multiply` argument all work now. What is refused should be what is
missing, so an unknown operator and a wrong operand count took their place.
191/191 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz, e2e 49, e2e2
concurrent, e2e3 16, e2e4 17, e2e6 72, e2e7 86, crud corpus unchanged at
201/90/196.
|
||
|
A.Shakhmatov
|
1c72aa8938 |
tests/spec: record the expression evaluator's spec from mongod
27 cases, recorded before a line of the evaluator is written, which is the
whole point of having built the recorder first: the expectations come from
mongod 8.3.7 rather than from what the implementation is about to do.
The corpus reads 1 pass / 26 fail, and the one pass is the unknown-operator
refusal M2 already answers correctly. Expressions are exercised through
`$group` because `_id` and the accumulator arguments are the only expression
positions that exist until `$addFields` and `$project`'s computed fields land;
testing them anywhere else would be testing a stage that is not there.
Ten things the recording settled that guessing would have got wrong:
$add over a missing field or null null -- not an error, and not 0
$add over a string error 7157723
$divide by zero error 4848401
$mod of -5 by 4 -1, the dividend's sign
$lt of a number and a string true, canonical type order
$and over -5 truthy
$not of a missing field true
$switch, no branch and no default error 40069
two operators in one expression error 15983, *not* $group's 40238
$subtract with one operand error 16020
The six error codes are in `ErrorCode` already, so the evaluator's refusals and
its runtime failures have somewhere measured to land. The evaluator itself is
the next commit and is not started.
191/191 unit tests, crud corpus unchanged at 201/90/196.
|
||
|
A.Shakhmatov
|
76afa75efe |
commands: the $group accumulators
Nine of them -- `$sum`, `$avg`, `$min`, `$max`, `$first`, `$last`, `$push`,
`$addToSet`, `$count` -- and the corpus goes 9 pass / 10 fail to 18 pass /
1 fail. The one left is the compound `_id`, which needs the expression
evaluator.
Landed before that evaluator, against the tier order the design review set
out, and the corpus is why: every one of these takes a single value per
document, a path or a constant, so nine of its ten failures turned out to be
reachable without one. `classify_expr` already produced exactly that value.
What the recording caught, which is the argument for measuring expectations
rather than writing them:
- `$avg` over a group with no numeric value is **null**, not `0`. A divisor
that counted documents rather than numbers would pass every test anybody
would think to write by hand, and be wrong on the one group that matters.
- `$min`/`$max` compare across types in canonical BSON order, so the maximum
of `30`, `7` and `"not a number"` is the string.
- `$push` skips an absent field but would push an explicit null, so "resolved
to nothing" and "resolved to null" cannot be the same value internally --
which is why the accumulators take `?bson.Value` and not `.null`.
- `$first`/`$last` follow input order, including when the value is absent:
`$last` of a missing field is null, not the last present one.
`AccState` is one struct rather than a union: the fields are small and every
site already switches on the kind, so a union would add a tag test where a
switch was going to be anyway. Its arrays are the gpa's, the values inside them
the reply arena's -- they outlive the group and travel with the documents.
`numeric_value` is the int32-or-double narrowing MongoDB reports, shared now
between the accumulators and `cmd_aggregate`'s count fast path. It was written
twice before; a divergence between them would make `countDocuments` disagree
with the pipeline it is a shortcut for.
`$avg` and `$push` came out of the Tier 0 refusal test, replaced by
`$stdDevPop` and `$mergeObjects`. The refusal is a property of what is missing
rather than of a list, and the test should read that way.
191/191 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz, e2e 49, e2e3 16,
e2e4 17, e2e6 72, e2e7 86, crud corpus unchanged at 201/90/196.
|
||
|
A.Shakhmatov
|
3044a38d1c |
tests/spec: an aggregation corpus, recorded from mongod
M2.5's gate, built before the milestone it gates -- the same order that put
`expectEvents` before the free list in M1 and Tier 0 before everything in M2.
`mongodb/specifications` has no aggregation suite, which is amendment A6's
central finding, so this milestone has to bring its own. The hazard in a corpus
we author is obvious and fatal: it can encode our own bugs as expectations and
then agree with us forever. So the split is enforced by the tooling.
`sources/*.json` holds documents and pipelines and nothing else; `record.js`
asks a real mongod 8.3.7 what each pipeline answers and writes the unified-
format file from the reply. Inputs authored, expectations measured -- the
discipline that corrected three assumptions in M1's session work and every
error code in M2, where the alternative would have shipped both times.
No second runner. `run.js --suite-dir` points the existing one somewhere else,
so the entity model, the matchers, the skip accounting and `expectEvents` come
for free; a second runner would drift from the first exactly where it mattered.
`--scorecard` is refused with `--suite-dir`, because `scorecard.txt` is the crud
corpus's record and the milestones are compared against it -- writing it from an
unrelated run would replace that record silently.
Errors record the code and not the message: message text is mongod's to change
between releases. Group pipelines end in a `$sort`, because group output order
is unspecified and a case depending on it would fail for the wrong reason on
either server.
The first source covers `$group`: nine accumulators including the edge cases
that decide an implementation -- `$avg` over a group whose values are not
numbers, `$min` of a field no document has, `$push` skipping a missing field,
`$first`/`$last` against input order, grouping on an array, a compound `_id`.
Where it starts, run against the M2 tip:
group-accumulators.json 9 pass 10 fail 0 skip
The nine include the four refusals M2 added, which answer with mongod's own
codes -- so the corpus already confirms that half. The ten are the milestone.
The crud corpus is unchanged at 201/90/196.
|
||
|
A.Shakhmatov
|
8ebeb9d4ec |
commands: $out and $merge, written by the dispatch epilogue
The seven reachable failures of M2, and the first commit of the milestone to move the scorecard: 194/97/196 -> 201/90/196, with `aggregate-*.json` going 9 pass / 13 fail to 16 pass / 6 fail. The seven that moved are exactly the seven priced as reachable, and the six that remain are exactly the six attributed to M2.5 ($addFields, the expression engine), M4 ($listLocalSessions) and M8 (collation). Both stages write to a collection the pipeline is not reading, and three things stood against doing that in the handler: `aggregate` is a `.read` command, dispatch takes locks from a static table keyed on the command name before the handler runs, and `Collection.lock` allows exactly one collection lock at a time. So the handler computes the output under the locks it has and leaves it in `Context.pending_write`; the epilogue applies it with nothing held, beside the commit and the checkpoint already there. The `.read`/`.write` contract is amended in its own comment rather than quietly broken. `pending_write` is cleared at the top of every dispatch, so a handler that errors before setting one cannot leave the previous command's write to fire. A failed write replaces the pipeline's `ok: 1` with the failure, because a client told the aggregation succeeded would believe the collection had been written. What the stages do not implement is refused, not ignored: `$merge`'s `whenMatched`, `whenNotMatched`, `on` and `let` all select behaviour this server does not have, and a `whenMatched: "fail"` that silently merged would be the same lie Tier 0 spent three commits removing. Codes measured against mongod 8.3.7. `$out` and `$merge` answer byte-identically to it on both the replace and the upsert case. NOT ATOMIC, and said out loud in the code rather than left to be discovered. mongod replaces an `$out` target atomically; this engine has no cross-collection atomicity and no rename to build one from, so a crash between the drop and the last insert leaves the target holding part of the new output where MongoDB would leave the whole of the old. The fix is write-to-temp-and-rename and rename is a command that does not exist here. The test's mutation is the argument for the epilogue in one line: apply the write inside the `$out` branch and it deadlocks rather than fails. 191/191 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz, e2e 49, e2e2 concurrent, e2e3 16, e2e4 17, e2e6 72, e2e7 86. |
||
|
A.Shakhmatov
|
a748a3d08c |
db/pager/tests: cleanup pass over the free list
No behaviour is meant to change and the gate confirms it: 1.94x / 679.2 MB
reclaimed / 9 rebuilds and 2.46x / 934.0 MB / 13 on the two 16 KiB lines,
0.0 MB on the 200-byte line, all identical to the numbers recorded for them.
Deduplication. `pages_for` was written in db.zig and again in pager.zig and
twice more inline; there is now one, public, and the two pre-existing copies
call it. `pages_per_map_align` replaces three hand-rolled `map_align /
page_size`. `SlabRun.window_first` was a stored field that could never legally
disagree with `first` and was maintained by hand at two sites -- now a method.
`keep_piece` re-derived `SlabRun.window_count` character for character; it
calls it. `insert_run` scanned linearly for a position `run_of` binary-searches
for, which made loading a fragmented catalog quadratic; both now go through one
`run_lower_bound`. Freeing a run's window map was written four times; one
helper. The 20% rebuild share was stated in `note_compact` and again in
`wants_rebuild`, with a comment arguing at length that they must be the same
number -- `worth_rewriting` makes that structural.
Efficiency. The identity assert in `reclaim_windows` called `dead_located()`,
an O(every window) walk, and `assert_msg` is live in ReleaseFast -- so it
doubled the scan the reclamation was about to make (2.75 MB streamed twice per
reclaiming checkpoint at the 21 GB the gate targets). `dead_located` is now a
maintained counter, the check is O(1) in every build, and the scan cross-checks
it while it is there. `SlabRun.full` lets a run with nothing to give be copied
without its counters being read at all, so the common case is O(runs) rather
than O(windows).
The pager's two allocation policies were hand-copying the claim step, and the
copy had already lost two of the three preconditions -- `alloc_slab_run` never
checked `pages <= reserved_pages`. Both now go through `claim_locked`.
`reclaimed_bytes` moves from Collection to Engine, beside `compactions`, which
is how it is read and the only place it can be honest: a life-of-the-process
total must not lose a dropped collection's share. Both join `Counters`, so
`slab_stats` stops opening `counter_lock` by hand.
The ownership assertion in `write_catalog` was gated on `is_test or Debug`, a
predicate nothing else in the codebase uses, which left the one silent failure
this design can produce unchecked in ReleaseSafe. It is now `!= ReleaseFast`,
the line `protect_stable` already draws. Measured: no change to the suite's
runtime.
Altitude. `note_checkpoint` was called from exactly one place, the tail of
`upsert` -- so a delete armed no checkpoint by any route, which is why
reclamation only ever ran when the *rebuild* trigger fired and the rebuild then
reset the window map it would have used. `remove` and the TTL sweep arm one
now, next to the `note_compact` calls that were added for the same omission a
milestone ago. `compact`'s leading checkpoint stays, demoted in its comment
from the mechanism to the local ordering it actually guarantees.
serverStatus reports `allocTailBytes`/`freeReadyBytes` instead of page counts,
so the harness stops hard-coding 4096 -- the kind of constant this milestone
was blindsided by once already.
tests/e2e/churn.js: `deleteMany({_id: {$in: [5000 ids]}})` exceeded
`index.max_combos`, so the planner refused the index and every delete became a
full collection scan re-filtering each document against 5000 members. That was
the entire runtime of the harness. One delete spec per id instead: the 40k x
16 KiB gate goes 57 s -> 6 s, and the 150k x 200 B line 483 s -> 3 s, with
identical output. Also: the per-round `countDocuments` is gone (the harness
knows the count), and the server log is a bounded ring rather than a rope that
grows with everything the server ever said.
Reverted from the review: reusing one MongoClient across the startup poll. A
client whose first connect fails tears its topology down and every later
command on it fails identically, so it turns "not up yet" into "never comes
up" -- it broke the first run. The reason is now a comment.
187/187 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz, e2e 49, e2e2
concurrent 2 and the crash pair, e2e3 16, e2e4 17, e2e6 72, e2e7 86,
crash-fuzz 60 cycles.
|
||
|
A.Shakhmatov
|
1491a47479 |
plan/results: the M1 churn numbers
Amendment A5 and the `[M1.1]`/`[M1.2]` blocks. What they record is a result with two halves, and the value is in keeping both: The mechanism works. 934 MB reclaimed over the update run, occupancy at 1.06-1.26x its live data, and the counters that say so are in `serverStatus` rather than inferred. The ratio did not move. 1.94x delete-heavy and 2.46x update-heavy, identical to the end-of-Stage-2 binary measured with the same harness. `file / live` is a high-water mark because the data file never shrinks, and the mark is set in the first round by the one thing reclamation cannot avoid: a rebuild needs a whole second copy of the live data before the first can be freed. So ~2x is the floor of a rebuild-based design and no threshold reaches it -- rebuilding earlier lowers the garbage term and nothing else, rebuilding later raises it. The plan said in advance what to do if this happened, which was to write it down rather than tune, and to name incremental compaction through a doc-id-to-offset indirection layer as the successor. Recorded, with its cost: a second copy-on-write B+tree per collection, a second random read on point lookup, and it undoes A3. A second lever is named that the plan had not: 52% of the steady-state file is space the database owns and is not using, so returning it to the filesystem is worth more here than reclaiming harder. It needs the file never to shrink below what the fallback generation references, which is its own crash-safety pass. D7.4's 1.65x for the delete line is corrected to 1.94x, and the correction is the harness rather than a regression -- the same 1.94x comes out of the binary that predates any of this work. The old ad-hoc version sampled ids to delete blindly, which re-picks dead ones, so it deleted fewer documents than it inserted and measured a collection that was quietly growing. The update line reproduces D7.4 exactly, 2.46 against 2.47. 200-byte documents reclaim nothing, exactly as forecast, and the forecast being written down beforehand is what makes that a result instead of a disappointment. 3.93x on both binaries. Also noted, because the number invites misreading: at that document size the two index trees are comparable to the documents themselves and the file is already 2.18x before any churn -- index structure, not slab garbage. |
||
|
A.Shakhmatov
|
8f63c6df70 |
tests/e2e: the churn gate, as a committed harness
D7.4 was the only block in `tests/e2e/results/m0-gates.txt` without a `reproduce:` line. The numbers were real and the harness was not committed, so the one measurement the whole free-list decision rested on could not be re-run against a change. This is that harness. Self-contained like `e2e6.js`: it spawns its own server on a fresh database. Two modes, delete-and-refill and repeated update, over `--docs` documents of `--doc-size`, with `--index` to put index maintenance inside the churn rather than beside it. Three things it does that the ad-hoc version did not: Live bytes are computed here, from the serialized size of one document, rather than read off the server. That is what makes a run against an older binary comparable -- and the first thing this harness was used for was measuring the pre-Stage-3 binary, which has no `multifora` section at all. Deleted ids are sampled from the ids actually live. Sampling blind from the id space re-picks dead ones, so a round deletes fewer documents than it inserts and a supposedly flat-live measurement quietly grows. The first run of this harness ended with 3211 documents where it should have had 2000. And it prints `inUse` beside `ratio`. The data file never shrinks, so `file / live` is a high-water mark and cannot come down however well reclamation works; `inUse` is `(allocTail - freeReady) / live`, which is what the database is actually occupying. On the update line those two read 2.46x and 1.06-1.26x for the same run, and the difference between them is the whole finding. A fixed seed, so two runs churn the same documents in the same order and a difference between them is the code rather than the dice. `--target x` fails the run above a ratio, for use as a gate; without it the harness measures and reports. |
||
|
A.Shakhmatov
|
6d5c860e11 |
tests/spec: an event's command is compared in the shape it was sent
A command-monitoring event hands over the command as the driver holds it in
memory, and that is not always the shape it puts on the wire: a sort is a JS
`Map` (driver lib/sort.js). `Object.keys` on a Map is empty, so the matcher
reported every key of an expected sort as missing from a command that in fact
carried it -- five cases, all of them the runner's fault and none the
engine's.
This one is worth the paragraph because of how well it hides. EJSON serializes
a Map exactly like a document, so `MFDB_DUMP_EVENTS` prints
`"sort":{"_id":1}` next to a failure that says `sort._id` is missing, and the
dump -- the tool built for exactly this triage in the commit that added the
buffers -- reads as evidence that the matcher is wrong about something else.
It took `Object.keys(formatSort({_id: 1}))` returning `[]` to see it.
Converted for the comparison only, and at every depth, since a sort also
appears inside `updates[i]`. `match` stays a plain reading of the spec's
Evaluating Matches with no driver knowledge in it.
Mutation-checked: pass the event's own value through and
findOne.json "FindOne with filter, sort, and skip" goes red again with the
original message.
189/102/196 becomes 194/97/196.
|
||
|
A.Shakhmatov
|
548c882d47 |
commands: the wire version says what the version string says
`buildInfo` has always reported 4.4.0 and the handshake has always reported maxWireVersion 8, which is 4.2. A driver believes the wire version: it refused client-side to send `hint` on an unacknowledged delete or findAndModify (the error is "only supported on MongoDB 4.4+", raised without a round trip), and withheld `comment` from getMore, listCollections and listDatabases (lib/operations/get_more.js:43 and its neighbours). Both are things this engine handles -- the acknowledged hint suites pass, and getMore ignores fields it does not know -- so the effect was purely the number disagreeing with itself. The 8 was not arbitrary. The comment above it tied it to omitting `topologyVersion`, which is what keeps a driver off the streaming hello protocol we do not implement -- a real bug, once visible as Compass reconnecting every heartbeat. Checked before touching it, in the driver rather than from memory: `useStreamingProtocol` (lib/sdam/monitor.js:154) returns false whenever `topologyVersion` is absent and never looks at the wire version at all. The omission is the whole mechanism; the wire version was a second line of defence that never existed. The comment now says so. A test asserts the two agree, so they cannot drift apart again silently, which is the actual defect here -- not the value. Spec suites 173/118/196 -> 189/102/196: 16 cases fixed, none broken. Ten are the unacknowledged-hint cases the previous commit uncovered, six are `comment` forwarding. 166/166 unit tests in ReleaseFast and ReleaseSafe, 82/82 fuzz, e2e 49, e2e2 concurrent 2 + crash pair, e2e3 16, e2e4 17, e2e6 72, e2e7 86, crash-fuzz 60 cycles. |
||
|
A.Shakhmatov
|
7f426cdd33 |
tests/spec: a collection entity gets the options it was declared with
`buildEntities` built every collection as `db.collection(name)` and every
database as `client.db(name)`, dropping `collectionOptions` and
`databaseOptions` on the floor. 15 collection entities declare a
`writeConcern`, 7 a `readConcern`, one a `readPreference` -- and the 15 are all
`{w: 0}`, so every "unacknowledged write" case in this corpus has been running
an acknowledged write against a driver that was never told otherwise.
They passed anyway, because an acknowledged and an unacknowledged write of the
same document produce results a `$$unsetOrMatches` expectation accepts either
way. Only the command on the wire distinguished them, and nothing was reading
the command until the previous commits. This is the first thing the event
assertions found, and it is a fair answer to what they cost.
Option documents are unwrapped from their BSON types on the way to the driver.
The suites are parsed with `relaxed: false`, so `{w: 0}` arrives as an Int32
and the driver gates `writeConcern.w` on `typeof w === 'number'` -- the same
trap NUMERIC_OPTIONS already documents for operation options, and a silent one:
the option would simply not apply. Wholesale unwrapping is safe here in a way
it is not there, since these are settings the driver consumes rather than
values an assertion compares. An option key outside the spec's
`collectionOrDatabaseOptions` set is reported unsupported rather than ignored,
which is the lesson of the bug itself.
159/132/196 becomes 173/118/196. 14 cases fixed, none broken.
The other 10 unacknowledged cases now fail differently, and that is progress
of a sort: with `w: 0` actually applied, the driver refuses client-side to send
`hint` on a delete or findAndModify to a server older than 4.4. This engine
reports itself as 4.4.0 with maxWireVersion 8, and 4.4 is wire 9. That
inconsistency is ours, it is the same one behind the `comment`-on-getMore
failures, and it gets the next commit.
|
||
|
A.Shakhmatov
|
bb8cdd964b |
tests/spec: the scorecard no longer disclaims expectEvents
The disclaimer was accurate for as long as it stood -- events were not read,
so `pass` was an upper bound and saying otherwise would have been a lie about
the number. It is now a lie in the other direction, so it goes, replaced by
what is actually true: events are compared exactly, in number and in order,
which is what makes a pass mean the engine answered correctly *and* was asked
the right question. The header says plainly that scorecards recorded before
this are not comparable, and enumerates what is still skipped inside events
rather than leaving "asserted" to be read as "asserted completely".
Two facts in the docs had gone stale and are corrected here because this is
the commit that rereads them:
- README said `--op-timeout-ms` defaults to 3 s. It has been 10 s since the
commit that explains, at length and directly above the constant, why 3 s
was wrong. A stale number in exactly the place that warns against
tightening it is worse than no number.
- `MAX_SCHEMA` is [1, 24]; the comment above it still claimed 1.0-1.9.
Totals unchanged at 159/132/196 -- this commit only rewrites prose, and the
scorecard is re-recorded so its header matches the runner that produced it.
|
||
|
A.Shakhmatov
|
54ad124c18 |
tests/spec: a CSOT-rewritten maxTimeMS cannot be asserted
Every client entity is built with CSOT `timeoutMS` (OP_TIMEOUT_MS, 10 s), and CSOT overwrites `maxTimeMS` on each command with what is left of that budget. An expectation of `maxTimeMS: 6000` therefore meets the harness's 10000, and no amount of engine correctness would change it. Reported unsupported rather than failed: a FAIL is a claim about the engine, and this is a claim about the runner. Refused unconditionally when an expected command mentions `maxTimeMS`, not only when the two values differ, so it can never become a pass by coincidence. Exactly one case in the corpus asserts it -- estimatedDocumentCount.json, "estimatedDocumentCount with maxTimeMS" -- so the whole cost of the hatch is one case, which is why it is worth taking instead of dropping `timeoutMS`. That option is not open anyway: `timeoutMS` is what replaced the outer race that once turned ~190 good cases into phantom timeout FAILs. This is the only escape hatch in the runner. Everything else is either an honest FAIL or an enumerated unsupported feature. 159/133/195 becomes 159/132/196: one case, fail to skip, and nothing else moves. |
||
|
A.Shakhmatov
|
97e3e3a556 |
tests/spec: assert expectEvents
The headline is not the delta, it is that `pass` changed meaning. 354 of the
487 cases declare `expectEvents` and until now the runner read none of them,
so a case could send the wrong command entirely and still be counted a pass
as long as the *result* came back right. The old column was an upper bound by
construction. 193/99/195 becomes 159/133/195, and the two numbers are not
comparable.
Two rules decide how far the assertion reaches, both taken from the spec
rather than from what would be convenient:
- `command` and `reply` match as *root* documents
(unified-test-format.md:1020-1022, :1037-1039). The driver hangs `lsid`,
`$db` and `maxTimeMS` off nearly everything it sends; as nested documents
essentially the whole corpus would fail on keys no expectation was ever
written to mention, and the number would say nothing.
- the event list is exact in number and order, not a prefix
(unified-test-format.md:3088-3091). 23 cases expect an empty list and a
prefix rule would pass every one of them without looking.
The assertion runs after the operations, so a wrong result is still reported
as a wrong result rather than being masked, and after the listeners are
disabled, so the teardown's own commands cannot reach the buffer.
`cmap` and `sdam` event types, `ignoreExtraEvents`, and any event field beyond
`command`/`reply`/`commandName`/`databaseName` are reported unsupported at the
point of assertion. None occurs in this corpus -- all 354 blocks are
`eventType: command`, carrying 349 `commandStartedEvent` and 6
`commandSucceededEvent` -- so nothing is being quietly waived.
All 34 newly-failing cases, triaged. Not one is a wrong answer from the
engine; every one is a command the driver never sent:
- 22x `command.writeConcern: missing` -- runner gap, and the sharpest thing
this commit found. `buildEntities` drops `collectionOptions` on the floor,
so `writeConcern: {w: 0}` never reached the driver and every "unacknowledged
write" case in the corpus has been running an acknowledged write. They
passed because the results of the two agree. This is precisely the class of
error the instrument was built to find, and it was invisible to the result
column.
- 5x `command.sort.<key>: missing` -- runner gap. The driver holds a sort as
a JS `Map` (lib/sort.js), so `Object.keys` on it is empty and the matcher
reports every expected key as absent. Measured, not guessed: EJSON prints a
`Map` exactly like a document, which is why the dump looks correct.
- 4x `command.bypassDocumentValidation: missing` -- unclassified. The option
is absent from the wire for the `false` cases; the driver only forwards it
when true on some paths (lib/operations/find_and_modify.js:19), and whether
the runner also drops it has not been established.
- 2x `command.comment: missing` on getMore -- server gap, most likely. The
driver gates it on `maxWireVersion >= 9` (lib/operations/get_more.js:43)
and this engine advertises 8 while reporting itself as 4.4.0, which is
wire 9. The inconsistency is ours.
- 1x `command.maxTimeMS: expected 6000, got 10000` -- the CSOT rewrite, dealt
with in the next commit.
Each of those gets its own commit, and none of them is fixed here: a check and
the fix for what the check caught do not belong in one change.
|
||
|
A.Shakhmatov
|
6560aec915 |
tests/spec: buffer command-monitoring events per client entity
Plumbing only: a client entity that declares `observeEvents` now gets `monitorCommands` and a buffer, and nothing reads the buffer. That is the point of splitting it out -- the totals not moving *is* this commit's test. Command monitoring changes how the driver builds every command it sends, and if that alone shifted a result there would be no way to tell it apart from the assertions landing in the next commit. 193/99/195 before, 193/99/195 after, 175/175 files, 0 errored. The rules the buffer already enforces, so that the next commit is only about comparing: `ignoreCommandMonitoringEvents` by command name; sensitive commands dropped unless `observeSensitiveCommands` says otherwise, with `hello` and legacy hello inferred sensitive from the driver having redacted them to empty documents (unified-test-format.md:3070-3075). Neither fires on this corpus -- 136 client entities observe `commandStartedEvent`, 6 also `commandSucceededEvent`, and not one sets either field -- but a rule that only exists where it is exercised is a rule that will be missing when M7 brings auth. `cmap` and `sdam` observations are collected by nobody; a test that goes on to assert them is reported unsupported where it asserts, not where it declares. Two things about placement, both load-bearing. Listeners are attached after `connect()`, so a client's own handshake is not in its own buffer -- measured rather than assumed: with the buffers dumped, find.json's five cases show exactly `find`, `getMore`, `getMore` and nothing else. And they are disabled after the operations and before the outcome check (unified-test-format.md:3081), plus again unconditionally in the teardown `finally`, because the outcome check and the teardown both issue commands and a buffer still growing through them would make the assertion a function of the harness rather than of the engine. `MFDB_DUMP_EVENTS=1` prints each case's buffer. That is how the handshake question above was settled and how a failing event assertion will be triaged. |
||
|
A.Shakhmatov
|
afa5c6ef9d |
tests/spec: $$unsetOrMatches does not change root-ness
`special()` passed a hard `false` for `root` into its recursion, so a value
standing behind `$$unsetOrMatches` was matched as a nested document even when
it sat at the top of an `expectResult`. The spec says the opposite in so many
words -- "This operator does not influence whether or not an actual document
value is considered a root-level document" (unified-test-format.md:2873, and
:2821 for `$$matchesEntity`) -- and that distinction is the whole of the
extra-key rule: only a root document may carry keys the expectation does not
mention.
`root` now threads through `match` -> `special` -> the recursion. From under a
key it is always false, which is what it already was; from the top level it is
whatever the caller had.
25 cases go from FAIL to pass and none moves the other way. Every one is the
same shape -- an `expectResult` of `{$$unsetOrMatches: {acknowledged: false}}`
against a driver write result that also carries its counts, or the
`insertedId`/`insertedIds` forms of the same thing -- and every one was the
runner failing a result the engine had got right. 168/124/195 becomes
193/99/195; the scorecard is rewritten here so the delta belongs to this
change alone.
Mutation-checked: put the `false` back in the `$$unsetOrMatches` arm and
bulkWrite-deleteMany-hint-unacknowledged.json returns to 0 pass, 2 fail. The
`$$matchesEntity` arm is the same one-word change on the same sentence of the
spec, but the crud corpus does not use that operator once, so it rests on the
spec text rather than on a red test.
Two consecutive full runs, both 193/99/195, 175/175 files, 0 errored, no
lingering timers.
|
||
| f2844e7894 |
cursors: server-side cursors for find, aggregate and the listing commands
Every reply came back in a single batch with `cursor.id = 0`, `getMore` was a
stub answering an empty `nextBatch` on the literal namespace `test.$cmd`, and
nothing read `batchSize`. That caps the useful collection size at what fits in
one 48 MiB message, which is the opposite of the tens-of-GB target and the
reason M0 made whole-index scans stream: the streaming candidate generator
existed with no consumer that could suspend.
## What a cursor is allowed to remember
A cursor holds no lock between requests, so everything it saves has to survive
arbitrary concurrent mutation. Nothing here is a pointer, and the two things
that look like stable addresses are not: `reset_tree` re-creates node ids 0 and
1 as different nodes, and `rebuild_collection` moves every document. Three
sources, chosen by query shape, each with a different memory contract:
- **stream** -- an index-ordered walk resumed from a `(key, off)` anchor plus a
`(leaf, slot)` hint. O(key) state, so this is what lets a cursor walk a
collection larger than memory. Survives a rebuild, because a repack changes
no key.
- **offsets** -- the matched slab offsets a narrowed plan already materialized,
8 bytes each. Killed by a rebuild with `QueryPlanKilled`, because those
offsets now name unrelated bytes.
- **buffered** -- canonical BSON copies, for a sort no index provides and for
aggregate/listing output. Depends on nothing, which is what lets a listing
hold a cursor over a `$cmd.*` namespace no collection backs.
`Collection.layout_epoch` and `Index.epoch` are the invalidation tokens, both
checked as error returns rather than assertions since a client reaches them by
keeping a cursor open across maintenance.
## Resume
`resume_forward`/`resume_reverse` are O(1) while the hint holds and fall back to
an exact-order band walk bounded by `resume_walk_max`. Without the hint, `seek`
lands at the *start* of an equal-key band, so `sort({status: 1})` over three
distinct values across 10M documents would cost ~5e10 comparisons to drain.
Two hazards found by draining a collection while writing to it, neither
predictable from reading the code:
- A deleted anchor must resume at its *band position*, or the rest of an
equal-key band is silently dropped -- most of the collection on a
low-cardinality index. Hence `band_index`.
- On a **unique** index a same-key entry can only be the anchor rewritten, so
resuming at it returned updated documents twice. Observed as duplicate `_id`s
while updating underneath a drain.
## Protocol
Measured against mongod 8.3.7 rather than recalled, which corrected three
assumptions: a bare `getMore` does *not* inherit the find's `batchSize` (4998 of
5000 documents come back), a namespace mismatch is `Unauthorized` (13) not
`CursorNotFound`, and `CursorInUse` is 143 not 12051.
`internalQueryFindCommandBatchSize` is 101, `cursorTimeoutMillis` 600000,
`clientCursorMonitorFrequencySecs` 4.
The rule everything follows is **never look ahead**: a batch that met its target
leaves the cursor open even when the source is in fact exhausted, so four
documents at `batchSize: 2` take three commands. `limit` acts as an EOF source,
which is what makes `batchSize == limit` close in one round trip. `skip` is
consumed once. `batchSize: 0` returns an empty batch with a live cursor.
Cursor ids are `(nonce << 20) | slot`, always positive. The nonce is not
decoration: without it a recycled slot serves one client another's documents.
Cursors are not connection-pinned, since the driver spec allows a `getMore` on
any connection to the same server; they end at exhaustion, `killCursors`, or the
idle sweep (a second monitor fiber, separate from the TTL one because the
cadences differ by an order of magnitude and a TTL failure must not stop
reclamation). The registry is fixed-capacity and evicts the least recently used
cursor, whose client sees the same 43 an idle timeout gives.
Fixed alongside, because cursors are what expose them:
- `listCollections` reported `"<db>."` with an *empty* collection part, which
makes the driver throw client-side -- so it would have broken the moment its
cursor stopped being id 0. Now `<db>.$cmd.listCollections`, as mongod uses.
- `count` ignored `skip` and `limit` entirely.
- `wire.end_message` now bounds a reply by the 48 MiB we advertise rather than
by `maxInt(u32)`; a reply past what we told the client to expect is not a large
reply, it is a desynchronized connection.
- Two `codeName` strings were wrong: 72 is `InvalidOptions` (MongoDB has no
`InvalidArgument`), and 40324 reports as `Location40324`.
## Verification
Unit 160/160 in ReleaseFast and ReleaseSafe; `tests/e2e/e2e7.js` adds 86 cursor
checks across five phases (batching/lifecycle/errors, streaming across churn,
aggregate+listings+count, expiry+capacity, restart) and is self-contained
because cursor behaviour is only observable with non-default flags. No
regressions: e2e 49, e2e3 16, e2e4 17, e2e2 2, e2e6 72. Spec 168 pass / 124
fail, +5 against the previous scorecard.
Mutation-checked, per the repo's second ground rule: `hint_slot + 1`, the
`band_index` off-by-one, both epoch bumps, the id nonce, the at-least-one-
document rule, and `stream_shape` returning null each turn the intended test
red. One claim was withdrawn rather than kept -- swapping `std.mem.order` for
`cmp_prefix` in the band walk changes nothing observable, so the comment now
says so instead of asserting a check that does not hold.
|
|||
| cd88e1a4d1 |
index/pager: place a split's new sibling positionally, and fix mmap growth alignment
Two bugs, both of which the crash fuzzer surfaced and neither of which any
existing test could see.
**A split put the new sibling in the wrong slot when separators repeat.**
`split_leaf` located the new right sibling with `separator_pos(node, key)`, a
search for the promoted key. That agrees with "immediately after `left`" only
while separators are distinct. When several children share one -- ten distinct
values across thousands of documents, so each value spans dozens of leaves --
`separator_pos` returns the slot after the *whole* equal-key run, which puts
the sibling at the end of that run while the leaf chain has it right after
`left`.
Parent child order then stops matching leaf chain order, and that is the one
thing a lookup cannot survive: `descend_lower` picks the last child of the equal
run, and `lookup_eq` walks forward from there over keys *smaller* than the one
it wants, stops at the first mismatch, and reports nothing. Every entry is
present, the chain is correctly ordered, `count()` is right -- and the query
returns empty. `crash-fuzz.js` found it after ~700 heavy cycles as
`find({k: 3})` returning 0 of 401 documents while every other key was exact.
Fixed by `child_slot_after`, which is positional by construction.
**mmap growth rounded with a non-power-of-two alignment.** Past 64 MiB the
growth chunk becomes a proportion of the current size (`mapped_pages / 8`),
which is not a power of two -- and `std.mem.alignForward` asserts that it is.
In safe builds that panicked; in ReleaseFast, where the assert is compiled out,
it computed `(addr + align - 1) & ~(align - 1)` with a non-power-of-two mask,
which can round *down*. A mapping shorter than intended is survivable, but a
mapping longer than the file is exactly what this function exists to prevent: a
store into a mapped page past end-of-file raises SIGBUS, which no error path
catches. `alignForwardAnyAlign` instead. Never noticed because no unit test grew
a pager past 64 MiB.
Also here, because both bugs were invisible rather than merely unfixed:
- `assert_indexes_cover_every_document` (db.zig) checks the index invariant
directly -- an index generates candidates and the full filter is re-applied to
those, so a missing entry is a missing query result nothing else detects.
- `Index.unreachable_key_count` counts keys present in the leaf chain but not
reachable by descending from the root, which is precisely the state above:
healthy by every other measure.
- `Index.dbg_root` dumps parent/chain agreement. Marked TEMPORARY; drop it once
the invariant checks have earned their keep.
- `crash-fuzz.js` now asks the same question without the index, so a failure
says whether the documents are wrong or only the index's answer about them,
and reports per-key totals so one lost leaf is distinguishable from an empty
index.
Verified: `zig build test` in ReleaseFast and ReleaseSafe, and seeded fuzzer
runs that previously reproduced the split bug.
|
|||
| 62caf9fefc |
tests/fuzz: tighten the listIndexes NamespaceNotFound comment
Comment-only cleanup: the six-line rationale restated the scenario twice (first-cycle crash at prefix 0 = kill during the first in-flight command on a fresh log) and echoed 'expected state' with 'exactly the case worth verifying'. Four lines keep all three points: real MongoDB answers NamespaceNotFound too, it is expected when nothing durable created the collection, and treating it as a harness error broke verification of that case. |
|||
| a9f625f82a |
tests/fuzz: listIndexes on a namespace the prefix never created is expected
`crash-fuzz.js` aborted with "harness error: MongoServerError: ns not found" whenever the surviving prefix contained no write that created the collection -- a kill during the first in-flight command on a fresh log. `listIndexes` on a missing namespace is NamespaceNotFound, which is what real MongoDB answers too, so the server was right and the harness treated a legitimate state as its own failure. Worse, it aborted the run instead of verifying that state, which is exactly the state worth verifying. Reproduces with `--seed 1234 --rounds 60` and is why seeded runs were unusable; `--heavy` happened to miss it. Confirmed against the previous commit before changing anything, so it is the harness and not the engine. Both seeds now pass 60 cycles. |
|||
| 6c7f1f2e77 |
tests/fuzz: crash-consistency fuzzer (crash-fuzz.js + README)
Black-box SIGKILL fuzzer for the M0 mmap+WAL crash story. Random write
workload through the official driver, kill -9 at a random point, reopen the
same log, verify the recovered state against an in-memory model:
- prefix invariant: recovered state == history[0..m) for some m in
[acked, sent]; every acked write durable, in-flight commands all-or-nothing
(group commit), nothing after them may survive
- always-opens (replay never refuses); index presence tied to the prefix and
find({k:v}) correctness (rebuild after replay); countDocuments
- unexpected server death (Zig panic, replay refusal) is a finding with the
server log; --verify-exec read-backs updates to separate execution bugs
from replay bugs; --no-kill for graceful-restart runs; --heavy passes a
1 MiB compact threshold to fuzz compaction/checkpoint windows
Deterministic via seeded PRNG; failures dump a repro artifact with the seed.
Mutation-checked: over-strict prefix check goes red on lost in-flight ops.
Observation recorded: small churn-heavy DBs can exhaust the pager's 64 GB
address-space reservation (data file grows in >=8 MiB compounding steps and
never shrinks without a rebuild), surfacing as DatabaseTooLarge on writes.
|
|||
| d597597a4c |
plan/results: record the two post-gate CRUD corrections
The gate results file said the replacement-style-update gap was found and not fixed, and PLAN said it was left alone. Both were true when written and are not now, so a reader would take the M0 scorecard for the current one. The M0 figures stay as measured -- they are the gate result -- with a pointer to tests/spec/scorecard.txt, which always holds the current number. |
|||
| 21489723a9 |
db: a write that changes nothing is not a write
`nModified` counted every write, so an update that altered nothing was reported
as a modification. MongoDB counts a document as modified only if applying the
update changed it, and writes no oplog entry when it did not: `$set: {x: 11}`
on a document already holding `x: 11` is matched and not modified. The spec
suite says it plainly -- `bulkWrite` with four updateOne operations expects
matchedCount 2 and modifiedCount 1.
Decided in the engine rather than the command, because that is where the
document is already serialized: the comparison is against the bytes that would
actually be stored, and it lands before the log append, so a no-op costs no log
record, no fsync, no slab bytes and no garbage. `Engine.replace` returns
`Written.modified` or `.unchanged` and `cmd_update` counts the first.
That exposed a second difference. A replacement keeps `_id` at the front, so
replacing a document with itself was a byte-level change whenever `_id` was not
stored first -- and it usually was not: the Node driver fills a missing `_id` by
assigning the property, which in JavaScript appends it, so `insertOne({name,
age})` reaches the server as `{name, age, _id}` and we stored it that way.
MongoDB moves `_id` to the front whatever order it arrives in. Now so does
`serialize_with_id`, for every document rather than only the ones whose `_id` it
generates. Visible to clients as `_id` coming back first, as it does from
MongoDB.
spec scorecard 161 pass / 131 fail -> 163 pass / 129 fail
bulkWrite.json 8 pass / 2 fail -> 10 pass / 0 fail
e2e.js 45 checks -> 49
No spec file regressed. Mutation: delete the byte comparison in `upsert`'s
`.replace` arm -- red on the log growing, on the garbage counters moving, and on
`replace` claiming `.modified`.
|
|||
| 53f88e6d3b |
update: replacement-style writes
`replaceOne`, `findOneAndReplace` and `bulkWrite`'s `replaceOne` all failed
with "bad update". `update.apply` rejected any update document whose first key
was not `$`-prefixed, so a replacement document -- which by definition has no
operators -- could not get through at all.
MongoDB decides on the first field and nothing else: `$`-prefixed means
operators, anything else means the document *is* the new content. An empty
document is a replacement too, and a legal one. `is_replacement` says which,
`apply_replacement` does the work, and because all three call sites already go
through `apply`, that one branch covers the update command, findAndModify and
the upsert builder.
What a replacement means, precisely:
- every field is replaced except `_id`, which is immutable and keeps its
position at the front, where it is stored and where the `_id_` index
descends on it;
- a replacement may restate the same `_id` but not a different one -- that
is `ImmutableId`, because otherwise a rewrite would silently change a
document's identity while the index entry kept the old key;
- when the target has no `_id` yet, the replacement supplies it. That is the
upsert path: `build_upsert_doc` seeds a document from the filter's
equalities, so `replaceOne({_id: 99}, {u: 1}, {upsert: true})` inserts
`{_id: 99, u: 1}` and not a generated ObjectId;
- a mixed document is refused from either side, rather than guessed at.
Two options on update specs are refused rather than ignored:
- `multi` with a replacement (FailedToParse). A replacement describes one
document; applying it to many would leave every match identical apart from
its `_id`.
- `sort`, a MongoDB 8.0 addition this server does not implement. Ignoring it
is the worst of the three answers -- `sort` chooses *which* match to write,
so the client would silently get a different document than it asked for.
spec scorecard 131 pass / 161 fail -> 161 pass / 131 fail
e2e.js 35 checks -> 45
Sixteen spec files improved and none regressed. The two `-sort` files briefly
did: they had been passing on their "server-side error" case, which our
"bad update" failure satisfied by accident, and passing for the wrong reason is
how a gap survives a scorecard.
Five mutations, each verified red: seeding the replacement from the old pairs,
dropping the `_id` comparison, dropping the `_id` a replacement supplies, and
removing the mixed-document guard from either loop.
Note `nModified` is still wrong for a write that changes nothing -- MongoDB
counts a document as modified only if applying the update altered it. That is
the remaining bulkWrite failure and is fixed next, separately.
|
|||
| 504179acd1 |
results: the M0 gates, measured
PLAN D7's six items, with the numbers and the command that reproduces each in
tests/e2e/results/m0-gates.txt. Unit tests green in both optimize modes, the
whole e2e matrix green, the spec scorecard byte-identical at 131/161/195, and
the large smoke run at the scale D7.3 asked for:
21.47 GB collection (1,310,720 x 16 KiB)
data file 21.75 GB (+1.3% over the documents)
log after the load 2.5 MB (checkpoints reclaim it)
kill -9 then reopen 0.5 s (0.5 s at 4 GB too -- flat)
RSS after reopen 237 MB (1.1% of the data)
count after restart 1,310,720 last document byte-intact
acked writes after kill 200/200
That is the milestone's claim, measured: an open costs the working set rather
than the size of the database. Before M0 the same measurement was 523 MB
resident for a 512 MB database, because recovering each document's `_id` meant
reading every document at open.
Two gates need reading rather than a tick, and m0-gates.txt says so where a
reader would otherwise take a tick for granted.
The churn gate settles at 1.65x live data (delete-heavy) to 2.47x
(update-heavy), flat, above the ~1.3x amendment A2 hoped for. Rebuild-only
reclamation cannot reach that: it needs a whole second copy of the live data
before the first can be freed. The gate existed to decide whether doc-level
free lists are needed after M0, and that is the answer.
Benchmark parity holds for every read and latency row inside the run-to-run
spread, and bulk insert regresses 24% (732 -> 555 MB/s), reproducibly across
three runs. Risk 1 as written: document bytes now reach the disk uncompressed
on top of the LZ4 log. createIndex improves 62% from the same change.
Three measurement bugs fixed while running the gates, because each would have
put a false number in the README:
- `compare-run.sh` measured "db on disk" as `du` of the log alone against
`du` of mongod's whole dbpath. It reported 20 MB for a 1 GB collection --
the documents had moved to <db>.data. Honest figure, measured: 914 MB of
allocated blocks against mongod's compressed 85 MB.
- `big.js` counted "compaction events" as "the log shrank", which is a
*checkpoint* now. It claimed 12 compaction rewrites during a pure insert
load, which has no garbage to compact.
- `big.js` labelled peak RSS "in-memory engine: docs live in RAM" and its
summary said the collection was held "fully in RAM". Both were true of the
engine this milestone replaced.
README: the storage section described an all-in-RAM engine; the comparison
table mixed one old run's body with three new rows; and `findOne({_id})` was
documented as a full scan for integer ids, which the ordered `_id_` index made
false (2 ms against 55 s for a scan of the same 21.5 GB collection). The table
is now best-of-three for both servers, with the measured variance stated, since
two runs of the same binary moved the sub-10 ms rows by 27-51%.
|
|||
| 9dda943f26 |
db: documents live in the data file
Collection's slab of malloc'd 8 MiB segments becomes extents in the data file,
and a document's offset becomes an absolute file offset. That makes `doc_bytes`
base + off instead of a binary search over segment starts, and it removes the
hazard the segment list carried: the mapping's base never moves, so a slice into
it cannot be invalidated by growth. phase8 records a dangling-slab-pointer bug
of exactly the shape this deletes.
`slab_reserve` now runs before the log append and `slab_append` after it, and is
infallible. It had no reservation before because appending to an ArrayList could
only fail on OOM; a file-backed slab can also fail on growth, and failing after
the record is durable would report an error for a write the next open produces
anyway.
The data file is still recreated empty on every open and the log still replays in
full, so nothing durable depends on it yet and open/close semantics are
byte-for-byte what they were. The watermark that turns it into a checkpoint comes
next.
--
One bug, and it is worth reading because the milestone keeps producing this
shape. Collection holds a `*Pager`, and `Engine.open` builds an Engine on the
*stack* and returns it by value -- so every pointer taken during replay dangled
the moment it was moved. It surfaced as SIGBUS, then as a corrupt hashmap in the
*second* engine of an unrelated test: nothing resembling its cause. The pager is
heap-allocated now, as `Index` already was, for the same reason.
--
Measured on one harness, 512 MB / 16 KB docs, before and after:
bulk insert throughput 742.6 MB/s -> 746.7 MB/s
createIndex({k: 1}) 26.8 ms -> 16.2 ms
countDocuments({}) 2.1 ms -> 1.1 ms
findOne({k: 500}) indexed 0.75 ms -> 0.53 ms
find({p: range}).count() 6.6 ms -> 4.1 ms
aggregate $group by k 5.8 ms -> 3.7 ms
insertOne (sequential) 0.20 ms -> 0.20 ms
Every row equal or faster. The regression this commit was scheduled early to
catch -- document bytes now reaching disk uncompressed on top of the LZ4 log --
did not appear at this size; bulk insert is flat and the reads gain from one
contiguous mapping instead of separately allocated segments.
What did *not* improve, stated plainly because it is the milestone's headline
claim: RSS is unchanged, 552 MB against 563 MB for 512 MB of data. It cannot
improve yet. Every open still replays the whole log and rewrites the whole slab,
so every page is touched and resident regardless of where it lives. "RSS =
working set" only becomes measurable once an open loads a checkpoint instead of
rebuilding, and the honest test for it is the multi-GB reopen in big.js, not this
microbenchmark.
|
|||
| 9390021b1e |
index/commands: stream whole-index scans; add a reverse leaf iterator
A whole-index read used to materialize every candidate before the caller saw
the first one. At the tens-of-GB target that is a list of every offset in the
collection -- ~160 MB for a countDocuments({}) over 20 million documents --
which defeats the point of moving storage to disk. Cursors are M1, but
*streaming a scan* has to exist now.
`Candidates` is the one loop candidates arrive through, whatever produced them:
a plan's materialized lookups, or the index read end to end. That keeps this
file's governing invariant -- an index only generates candidates, the full
filter is re-applied to every one -- in a single place. A narrowed plan still
materializes, because its multikey/$in dedupe genuinely needs the whole set and
is bounded by selectivity.
`RevIter` walks `Node.prev`, which has always been maintained and which nothing
had ever read: a descending scan materialized the whole index and reversed the
list. `find({}).sort({_id:-1}).limit(20)` becomes O(20).
The unfiltered fallback now walks the _id_ index instead of the docs map. That
is ordered rather than hash-ordered, and it does not depend on a structure that
is about to be deleted.
`Plan.full_scan()` refuses multikey indexes, since one document contributes
several entries there and a stream cannot dedupe what `search` did. The check is
currently redundant -- the planner refuses to order a multikey index anyway --
and is kept because the two guards protect different things. Stated precisely
in both places after checking: the commands.zig test reddens only when *both*
guards are removed, which is what that test actually pins.
--
This also broke e2e6's compaction check, and the fix there is the more
interesting half.
The check required peak/final > 1.4 and got 1.28. The final size was identical
to the byte (2,398,065 vs 2,398,064) -- compaction reclaimed exactly as before
-- and only the peak moved. Isolated to one variable: changing just the order
updateMany({}) walks its matches moves peak/final between 1.65 and 1.28, because
compaction can also fire from the once-per-second TTL monitor and whether one
lands inside the batch shifts the peak a long way while leaving the outcome
unchanged. The threshold was measuring the schedule.
Replaced with `peak > final`, which measures the shape instead: an append-only
log grows monotonically, so its maximum *is* its final size, and a file that
was ever larger than it ended can only have been rewritten.
Worth recording why the obvious alternative does not work. An absolute size
bound cannot distinguish a working compactor here: the payload is one repeated
character, so ~48 MB of records LZ4-compress to ~3 MB whether or not anything is
reclaimed -- with compaction disabled entirely the file still ends at 3.1 MB. I
first wrote the comment claiming that bound was the strong one, then measured it
and found the opposite; `peak > final` is what goes red.
|
|||
| 90de7820da |
tests/spec: MongoDB spec-test runner and the M0 scorecard
PLAN D2 makes the official specification suites the gate for command semantics; D7.6 asks for the harness to exist at M0 with a recorded baseline. This is that harness, pinned on both sides -- mongodb/specifications @ 615e0f9 and mongodb@7.5.0 -- because a scorecard is only comparable across milestones if a delta cannot be an upstream test change. It implements the unified format's Evaluating Matches algorithm as written, including the two rules that decide whether a pass is earned: extra keys are tolerated only in a root document, and numeric types compare flexibly. Anything unimplemented is a SKIP with a reason, never a pass, and the one assertion class not yet checked -- expectEvents, i.e. command monitoring -- is disclosed at the top of the scorecard so `pass` reads as an upper bound. First honest run: 131 pass, 161 fail, 195 skip over 175 files, zero timeouts. Getting there took four attempts, and the failures are documented in the README because each would have shipped a scorecard claiming a compatibility gap that did not exist. Two were genuine leaks in this runner (clients left open when a case timed out; clients registered for cleanup only after `await connect()`, plus abandoned cases still creating more). The third I misdiagnosed as machine load. The fourth attempt found the real cause: a leaked catalog lock in the engine, fixed separately, which alone accounts for the jump from 45 passes to 131. So the runner carries its own guards: per-operation CSOT timeouts so work is never abandoned, an active-handle census per file, an end-of-run tripwire for stray timers, a hard stop if the server dies rather than emitting hundreds of misleading ECONNREFUSED failures, and --skip/--limit for bisecting a run whose failures depend on position. The README states the rule plainly -- a long unbroken tail of timeouts is a harness bug until proven otherwise -- and the two commands that settle it. Also fixes bench-run.sh, which copied its report over bench-latest.txt unconditionally, including after a run that only warned -- so a degraded run could silently replace the baseline that PLAN D7.5 makes a milestone gate. |
|||
| e2c25a986b |
wire/server: honour moreToCome on OP_MSG requests
`Message.flags` was parsed and stored but never read. An OP_MSG request with
moreToCome set is fire-and-forget: the client will not read a reply. Sending one
anyway leaves it unread in the socket, so the next command on that connection
reads the previous command's reply and waits forever for its own.
This is not a corner case. Every unacknowledged write uses it, and the Node
driver sends `endSessions` with `writeConcern: {w: 0}` whenever a client closes
-- so an ordinary application that never asks for w:0 still hits it. Before:
insertOne({w: 0}) -> ok, acknowledged=false
countDocuments() (same conn) -> BSON element "cursor" is missing
The command still runs; only the reply is suppressed.
The e2e case pins maxPoolSize to 1, because with a larger pool the driver may
hand the next operation a different connection and hide the bug. It asserts the
connection still works afterwards, which is the part that matters -- not that
the unacknowledged write itself returned.
|
|||
| 411a380d38 |
commands: fix a remote invalid free in aggregate $sort
Present since at least
|
|||
| d4c9b04f21 |
rename project to MultiforaDB
Prose and benchmark tables use MultiforaDB; the binary, the CLI usage line, the log-message prefix and the default database file use multiforadb. Two consequences worth noting: - build.zig.zon's fingerprint is derived from the package name, so it had to change with it (Zig refuses to build otherwise). A consumer pinning this package by fingerprint needs updating. - the default --db path is now multiforadb.log, and getCmdLineOpts reports it as dbpath. An existing mongo-lite.log has to be passed explicitly with --db. The e2e harness abbreviated the old name as ML_; that is now MFDB_, including the documented ML_BIN override (MFDB_BIN) and the scratch file names. MD_ (mongod) is untouched. compare-run.sh spawned the server by absolute path under a sandbox/mongo-lite directory that no longer exists; that block already runs from tests/e2e, so it uses a relative path now. The archived reports under tests/e2e/results/ keep the old name: they record what the old binary measured. |
|||
| ac464f2b92 |
tests/e2e: iteration-to-iteration benchmark harness
compare-run.sh answers "how do we compare to MongoDB"; it says nothing
about whether a change made things better or worse than last week. Add a
harness that records each run and diffs it against the previous one.
bench-run.sh wraps compare-run.sh, adds a concurrent durable-write
comparison (concurrent.js: N clients each doing sequential insertOne with
{w:1, j:true}, exercising the group-commit path under real contention),
writes a versioned name<TAB>value report to results/bench-<timestamp>.txt,
and prints a diff of our numbers against results/bench-latest.txt.
Also:
- compare-run.sh polled with fixed sleeps, which are flaky once earlier
benchmark phases have warmed the machine; both servers now wait on a
real driver connection instead.
- a dispatch error only reached the client as a generic InternalError,
with nothing on the server side naming the failing command; log the
connection, command and error name before replacing the reply.
|
|||
| c8d547fef5 |
db/commands: acknowledged writes reach the disk again
Three defects, each of which made the database lose data that had already
been acknowledged, or answer a client with a malformed reply.
- Engine.commit decided a writer was "already covered" by comparing
log.end_pos with the position of the last completed commit. Under block
framing an append leaves its bytes in the log's open in-memory block and
does not move end_pos -- only sealing does. So once the first commit had
set committed_end = end_pos, every later write command found itself
covered and returned without sealing or syncing anything. A no-op
deleteMany followed by insertMany(50) was acknowledged with the file
still 16 bytes (its header) and lost all 50 documents on kill -9, which
is precisely what e2e2's crash pair does. Coverage is now decided by
sequence number, which counts records rather than bytes on disk.
- Compaction read the new log's end position before syncing it, but the
sync is what seals the open block, and the seal is what moves end_pos
past it. Appends after a compaction therefore started inside the
compacted file's last block and overwrote it, so those documents were
gone at the next replay: e2e6's phase 2 ended with 1000 documents in
memory and 996 after a graceful restart.
- cmd_find returned early on a missing namespace without putting anything
in the reply, so a find on an unknown collection arrived at the driver as
a response with no `ok` field ("MongoServerError: n/a") instead of an
empty cursor. The other commands' missing-namespace paths were fine.
Verified with the unit suite in ReleaseFast/ReleaseSafe/Debug, the split
fuzzer, all six e2e suites (e2e6 back to 72/72) and the kill -9 crash pair
-- none of which passed beforehand -- plus 13 kill -9 runs over 1/2/8
connections with 1200 acknowledged inserts each and nothing lost.
tests/e2e/results/phase7.txt records the benchmark with the fixes in place:
no regression against phase6 (bulk 739 -> 753 MB/s, updateMany 1.9 -> 2.0
ms, RSS 547 -> 546 MB), and concurrent durable writes now measurable at
7.1k/15.0k/21.8k docs/s over 1/8/32 connections.
|
|||
| ecd28d9b26 |
engine: decompose the global lock; cross-connection group commit (roadmap item 5)
The single engine-wide reader/writer lock is replaced by a lock hierarchy, so writes to different collections no longer serialize on one mutex: - Collections are heap-allocated, so their addresses are stable while a command holds a collection lock (the maps only store pointers). - A catalog rwlock guards the database/collection maps: shared for every command (so a concurrent DDL cannot mutate the maps underneath it), exclusive for create/drop/dropDatabase. Each collection has its own rwlock; the ordering is always catalog -> collection -> log lock, never two collection locks at once (TTL sweep and compaction take collections one at a time). - Command dispatch acquires the catalog + target collection locks for the handler's duration, resolving the collection (creating it for writes) under the catalog lock; create/drop upgrade to the exclusive catalog lock. - Appends never fsync. Each write command's epilogue releases the collection lock, then commits once (seal + fsync) with a leader/follower group commit: the leader waits for writers mid-append (a pending counter) so its seal covers them, and followers whose records the seal covered skip their own fsync. Every acknowledged write is fsynced before its reply (crash pair verified); an unacknowledged write may vanish and a reader may observe a write before its fsync — ordinary w:1 j:true semantics instead of 'the log describes >= memory'. - Compaction snapshots collections without the log lock (so a concurrent writer holding one can always finish its append) and retries when a writer appended mid-snapshot (detected via the record seq), then swaps under the log lock — no deadlock. The compaction trigger moved to the command epilogue and the TTL monitor. - Engine.dup_index moved to the collection (per-command error paths). Also lands two B-tree edge-case fixes driven by tests that were in flight: a churned leaf full of dead bytes no longer splits with an empty right half (the leaf is repacked before splitting, and an emptied node's page is fully free again), and a slot-count split with all large records on one side shifts records between the halves until the new record fits. Plus a randomised fuzz test over key sizes (src/fuzz_split.zig) and the two regression tests. Measured (tests/e2e/results/phase6.txt): no regression on the single-connection benchmark; concurrent durable-insert throughput ~5.1k -> 12.5k docs/s from 1 -> 8 clients, ~14.8k at 32. Verified: unit suite in all three modes, all e2e suites, the kill -9 crash pair. |
|||
| 570900a6ef |
storage: byte documents in a per-collection slab (roadmap item 4)
Documents live as canonical BSON bytes in a segmented per-collection slab (fixed 8 MiB segments keep capacity slack under one segment); the docs map holds flat offsets that stay valid across segment growth, and removed documents leave garbage bytes until compaction rewrites. The per-document ArenaAllocator and its second full Pair-tree copy are gone. The matcher walks the stored bytes directly, skipping by length any field the filter does not name (a new bson byte-walker: element_key, skip_value, read_value with borrowed leaves, get_at, and a borrowed spine parse). The byte matcher is differential-tested against the tree matcher on a corpus and shares its operator logic. Stored documents are never materialized on the scan path or in aggregate $match; $group reads group keys and sums straight off the bytes. Sort, projection, findAndModify, updates and index entry generation use a borrowed spine into the slab (or the byte collector, which also replaced collect_values in build_entries). The compaction threshold now counts uncompressed data volume, since a compressed log would otherwise never trigger. Measured (tests/e2e/results/phase5.txt): server RSS 1979 -> 539 MB (2.4x smaller than MongoDB; phase1 baseline 2.0 GB), range-scan 22.5 -> ~12 ms (parity, best run faster than MongoDB), proj 4.1 -> 3.4 ms, createIndex parity. Verified: unit suite in all three modes with zero leaks, the crash pair, e2e6, and the stress/spill programs. |
|||
| b4585106f1 |
storage: block-framed LZ4-compressed log (roadmap item 3)
The log is now a 16-byte file header (magic, version, codec, block target) plus a sequence of blocks. Each block keeps the pre-existing record framing unchanged, so Engine.apply_record does not change; records never straddle blocks (appends accumulate in memory and the block seals at ~256 KiB). The block header's integrity hash covers the stored payload bytes exactly as they sit on disk, so the decompressor only ever sees input already proven intact. Torn tails stay distinguishable from interior corruption exactly as before: a short read, an impossible length, or a hash mismatch in the final block truncates cleanly (later appends overwrite the garbage); a hash mismatch anywhere else is error.InvalidLog. The codec is a hand-rolled LZ4 block compressor/decompressor (~1.7 GB/s measured) with a per-block codec byte falling back to raw when compression does not help; the header keeps raw legal so zstd can be swapped in later. Zig 0.16 ships zstd decompression only, and deflate would cap writes below the insert rate. Engine.compact goes through the same Log API (deferred sync, one commit) and compresses for free; sync() seals the pending block before fsyncing, so the acknowledged-write durability semantics are unchanged (an unsealed block holds only unacknowledged batch records). Measured (tests/e2e/results/phase4.txt): db on disk 1025 -> 97 MB, now smaller than MongoDB's own compressed files; bulk insert 816 -> 722 MB/s (the accepted compression cost); reopen unchanged at 0.8 s. Verified: unit suite in all three optimize modes (new LZ4 round-trip, corrupt-block, and torn-tail truncation tests), the crash pair, e2e6 (kill -9 mid-write), and two full benchmark runs. |
|||
| 58914a69c3 |
index: ordered _id index (roadmap item 2)
Give every Collection an implicit _id_ index (a normal Index with keys
[_id: 1]) so _id equality, $in, ranges and sorts stop depending on the
docs-map hash or a full scan. Kept out of the secondary indexes list, so
listIndexes/dropIndexes/createIndex and the log format are unchanged (no
index_create record, no double listing) and e2e3.js passes unmodified.
Maintained in upsert through the same reserve-then-insert protocol as
the secondaries, removed in evict_doc, and rebuilt after replay by
build_all_indexes alongside them (never maintained mid-replay, so a
failed add can't leave the index under-approximating). index.plan now
takes it as a separate argument. Its keys are canonical
(bson.encode_key gives int32 1, int64 1 and double 1.0 identical bytes),
so the serialization-guarded docs-map fast path (plan_id,
value_fast_path_safe and friends) is deleted.
Measured (tests/e2e/results/phase3.txt): sort({_id:-1}).limit(20) 6.2 ->
2.4 ms (2.3x slower than MongoDB -> parity); integer/string _id point
lookups, $in and ranges verified against the tree. Unit suite in all
three optimize modes, the crash pair, e2e3/e2e4/e2e6.
|
|||
| 61fe952125 |
index: B+tree over the encoded keys (roadmap item 1)
Replace Index.entries (one sorted array) with a B+tree so writes into an already-built index stop being quadratic. Nodes are fixed 4 KiB slotted pages in a flat u32-addressed ArrayListUnmanaged(Node); records longer than a quarter page spill to an append-only overflow slab (BSON strings reach 16 MB). Leaves are doubly linked for ordered iteration; the flat node array stays one contiguous byte range for a later checkpoint. Insertion descends by separator and splits leaves/internals upward, promoting keys via a stable copy (a nested split can otherwise clobber the promoted-key scratch). Deletion does not rebalance: emptied leaves are unlinked and dropped from their parent, internal nodes may carry one child, and dead pages are abandoned in place (node memory peaks at the tree's peak size, exactly what the old array's capacity did). Lookups are lower-bound seeks plus leaf-chain band scans, so equal keys may span leaves freely. Bulk build (append_doc_entries + finish_bulk) sorts a staging array and packs leaves bottom-up. reserve_for now takes the built entries and reserves exact overflow bytes plus a worst-case node count, keeping insert_entries infallible after the log append. db.zig: TTL sweep now seeks the minimum-datetime encoded key and walks the contiguous datetime band, stopping at the cutoff or type change. Measured (tests/e2e/results/phase2.txt): updateMany 17.3 -> 1.8 ms (2.8x slower than MongoDB -> 3.7x faster), createIndex 62 -> 51 ms. Verified: unit suite ReleaseFast/ReleaseSafe/Debug (incl. the existing lookup_range and remove_doc differentials, plus a new incremental insert/remove differential against a brute-force model), the crash pair, e2e3/e2e4/e2e6, and dev stress tests for depth-2 splits, full drains, and spilled records through internal levels. |
|||
| 75e412a4af |
query/commands/wire: trim the scan and request paths
Matching allocated an ArrayList per filter field per candidate document,
on the process-wide allocator, to hold what is almost always a single
value. Candidates now collect into a stack buffer that spills to the heap
only for arrays: measured 15.7 -> 12.0ms on a 65,536-document range scan.
The OOM-propagation test moves with it. Its point is that a failed
collection must surface as an error rather than an empty candidate list,
which would make $ne and $exists:false report a match -- a wrong answer
rather than a failed one. That invariant still holds on the spill path, so
the test now uses an array long enough to reach the allocator, and a new
test pins the flip side: the common single-value match now completes
correctly even when the allocator always fails, because it never calls it.
Query operators were dispatched by a chain of up to fourteen mem.eql per
value per document, with $gt/$gte/$lt/$lte re-comparing the operator name
inside the loop over candidate values. Names resolve to an enum once per
filter field. Command dispatch likewise walked a 30-entry table comparing
strings; it is a comptime StaticStringMap now.
Each request built a fresh reply arena and handed its pages straight back.
One reply per connection, reset between requests, keeps them.
countDocuments() arrives as [{$match: F}?, {$group: {_id: <literal>,
n: {$sum: 1}}}], which the general path answered by materializing every
matching document and discarding them all. It is now recognized and
answered from a counting scan: countDocuments({}) 2.3 -> 1.5ms.
The detector is deliberately conservative -- grouping by "$field", summing
a field, an unmodelled accumulator or any extra stage all fall through to
the general path, since those need the documents themselves. A unit test
pins each accept and reject, and the whole count path was checked against
the general one through the real driver, including the shapes that must
not take it.
The filtered range-scan row does not move: it is bound by walking 65,536
documents that each live in their own arena, not by the matcher. That is
Phase 4 work.
Verified: 77 unit tests under ReleaseFast and ReleaseSafe, e2e
29/16/17/3/2, the crash pair, e2e6 72/72.
|
|||
| 552b916833 |
tests/e2e: record the Phase 1 gate measurement
Reproduce with bash tests/e2e/compare-run.sh 1g 16k. Without a stored baseline the phase gates in the plan are not checkable and the projected numbers are not falsifiable. |
|||
| 556ad7dc86 |
storage/db: XxHash3 record integrity, garbage-ratio compaction
Two streams of work land together: they are interleaved in storage.zig
and db.zig and only build as a unit.
Already in the working tree before this session:
- ReleaseFast as the default zig build (Debug was 10-200x slower)
- group commit: one fsync per write command instead of per document
- plan_id returned a pointer to a stack temporary; ReleaseFast read
garbage and silently broke findOne({_id: ObjectId})
- perf suite: big.js, compare.js, compare-run.sh, e2e6.js
Phase 1 performance work:
Record integrity hash CRC32 -> XxHash3. std.hash.Crc32 is the
table-driven byte-at-a-time Crc32IsoHdlc, measured at 408 MB/s against
XxHash3's 31 GB/s: 38us versus 0.5us on a 16 KiB document, which was
about two thirds of the entire bulk-insert cost. The record header
grows from u32 crc to u64 hash (header_len 20 -> 24), a breaking
format change. Bulk insert 260 -> 700 MB/s, reopen 1.1 -> 0.5s.
Compaction fsynced once per live document, because Log.open leaves
defer_sync false and compact never set it. It now issues one sync for
the whole rewrite, before the rename that publishes it.
Compaction triggers on the share of the log that is garbage
(live_docs/dead_docs, maintained at evict_doc, the single point where
a document dies) rather than on bytes appended. A fixed byte count is
wrong in both directions: a 1 GB bulk load holds no garbage at all yet
would compact ~64 times under the 16 MiB default, rewriting 1 GB each
time, while a small collection rewritten in place accumulates garbage
indefinitely without ever reaching the count. Pure inserts now never
compact, and the file stays near 1.25x the live data. Bulk load at the
default threshold: 41.6 -> 702.7 MB/s.
remove() never called maybe_compact, so a delete-heavy workload grew
the log without bound.
e2e6's compaction check required the file to bloat past 30 MiB before
being reclaimed, which encoded the old policy and failed on strictly
better behaviour (ends at 15.8 MB against ~12 MB live, was ~30 MB). It
now asserts the file ends near the live size and peaked well above it,
which does not depend on when the trigger fires. Sampling interval 50
-> 10ms: the operations now finish inside the old window.
Verified: 70 unit tests under both ReleaseFast and ReleaseSafe, e2e
29/16/17/3/2 checks, the crash-a/kill -9/crash-b pair, and e2e6 72/72
across three consecutive runs.
|
|||
| d90cde394c |
commands/e2e: drop topologyVersion from the handshake; rename to mongo-lite
Advertising topologyVersion in the hello reply is what tells a driver the server speaks the streaming (awaitable) hello protocol — in the Node driver it is the only condition checked. From the second heartbeat on, the driver then monitored with an exhaust hello (exhaustAllowed + maxAwaitTimeMS) and waited for a stream of replies carrying moreToCome. We answered once with the flag clear and went back to reading, so every heartbeat failed with "Server ended moreToCome unexpectedly", destroying the connection and clearing the pool. MongoDB Compass showed this as a connect/disconnect loop once per heartbeat. We do not implement streaming hello, so we must not claim to. Omitting the field keeps monitoring on the polling path, and agrees with the maxWireVersion 8 we report: streaming hello arrived in wire version 9. The existing e2e files all passed against the broken server — they issue their commands and exit before the second heartbeat — so e2e5 watches SDAM heartbeats on an idle connection instead. Also renames mongo-light to mongo-lite throughout (binary, log messages, docs, gitVersion). Unrelated to the fix above, but squashed in at request rather than left as a commit whose message described only the fix. |
|||
| 3c1ab6f656 |
index/db/commands/server: TTL indexes
createIndex({expireAt: 1}, {expireAfterSeconds: 60}) now deletes a
document once its indexed date is that many seconds old.
index.zig carries the option: Index.ttl (?i64), parsed from
expireAfterSeconds (int32/int64/integral double, within MongoDB's
[0, 2147483647]; 0 means "expire at the stored instant"), emitted by
spec_pairs and so persisted through the log and reported by listIndexes,
and compared by spec_equal — a same-name re-create with a different
expiry stays IndexOptionsConflict, as in MongoDB, since collMod does not
exist here. The bound keeps the value an int32 on the wire and makes the
emission cast unconditional. TTL is single-field only (a compound key is
error.TtlOnCompoundIndex); an absent option parses to null, so every
existing log record reparses unchanged.
db.zig sweeps: Engine.ttl_sweep(now_ms) walks each TTL index's entries
and deletes through the ordinary remove path, so an expiry is logged and
fsynced like any other write and holds across a restart. Expired ids are
duped before removal — remove frees the docs-map key that Entry.id
aliases — then sorted and deduped, because one document can be expired
by several entries (an array of dates expires on its earliest member,
which multikey expansion gives for free) or by several TTL indexes.
Selecting entries is a type test rather than a range lookup: bson
compare order ranks datetime above null, numbers and strings, so a
datetime upper bound would also select every value of a lesser type. A
sweep that deleted something checks the compaction threshold, since a
TTL-only workload never reaches the one in upsert.
commands.zig maps the new spec errors: CannotCreateIndex (67) for a
compound key or an out-of-range expiry, and InvalidIndexSpecificationOption
(197) for an expiry on {_id: 1}, which the idempotent _id no-op would
otherwise swallow.
server.zig runs the monitor as a member of the connection group, so the
existing group.cancel tears it down; it sleeps first, then sweeps under
the write lock, and logs rather than dies on a sweep failure.
--ttl-sweep-secs sets the interval (default 60, 0 leaves it unspawned).
Expiry is coarse by design, as in MongoDB: a document stays visible
until the next sweep, and a non-date value at the indexed path never
expires. Unit tests cover the spec round-trip and every rejection, the
sweep (inclusive cutoff, string/missing/future values untouched, second
sweep a no-op) and its survival of a reopen, and two TTL indexes over
one collection. e2e4.js exercises it through the Node driver.
|
|||
| adcf0014a9 |
docs/e2e: indexes out of 'not implemented'; e2e3 covers driver index APIs
README documents supported key patterns, unique/sparse/multikey behavior, planner rules (multikey two-bound range fallback, sparse/null bail, the _id fast-path guards), and v1 limits plus the two pre-existing issues the work surfaces (drop-collection resurrection, compact log_bytes). e2e3.js exercises createIndex/getIndexes/dropIndex/dropIndexes, the unique-constraint 11000 path, compound, sparse, and descending indexes through the official Node driver. |
|||
|
mongo-light
|
662df9b121 | tests/e2e: pin driver deps (package.json/package-lock.json); ignore node_modules | ||
|
mongo-light
|
c29c09d6e8 |
query: support bare regex filter values and array-index dot paths
The Node driver sends {field: /re/} as a BSON regex element (type 0x0B),
which the matcher previously only handled via the $regex operator form;
and dot paths with numeric segments (tags.0) were ignored because array
descent only recursed into embedded docs. Both are part of standard
MongoDB query semantics and were caught by the driver e2e suite.
server: unbounded Io async limit so the accept loop never wedges, and
treat header-read failures (client RST on pool teardown) as clean
disconnects. With the default cpu_count-1 limit, groupAsync's eager
fallback ran connection handlers inline on the accept-loop fiber once
that many connections were alive, stalling accept() and timing out
handshakes for further clients.
Add tests/e2e/: official driver CRUD, concurrency, and kill -9 recovery
suites (29 + 2 + 3 checks), plus unit tests for the query fixes.
|
