bf685aa6dee4bc1bb1cd5955f29bf74335bf5f25
61 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
A.Shakhmatov
|
21494469a5 |
db/index: a hashed key holds a hash of its value
M3's last row, step 4 of the index review's order. `{a: "hashed"}` was
refused with "invalid index spec" -- the right answer with the wrong code,
and the half of the row the review called honestly missing.
A hashed component is stored as a tag byte plus a 64-bit hash of the
value's *ordinary* encoded bytes. Hashing the encoding rather than the
value is what makes `{a: 5}` and `{a: 5.0}` land on the same entry for
free: `bson.encode_key` already normalizes every numeric type through
f128, because two values that compare `.eq` have to encode identically for
the tree to be a memcmp. One `encode_component` does it for entry
generation and both lookup paths, so the two sides cannot disagree -- a
component hashed on the way in and not on the way out would simply never
find anything.
Collisions are harmless, because this file's governing invariant is that
an index only generates candidates and the full filter is re-applied to
every one. The single place that would not survive one is uniqueness,
which is why `unique` is refused (16764) rather than approximated.
The planner is the mirror of the partial rule and just as conservative:
equality only. A range or a sort over a hashed component would read a band
of leaves ordered by hash, which is an arbitrary set of values, so both
are declined and the query scans. That is what leaves `find({a: {$gte:
5}})` and `find({}, {sort: {a: 1}})` correct.
The catalog needs no new field. `write_index_catalog` has always written
one byte per component and that byte has only ever held 0 or 1, so a third
value costs no format change and `catalog_version` stays 1. That is a
departure from the review, which guessed at a sixth flags bit: hashed
belongs to a *component*, and a compound index may hold one beside range
ones.
Measured on mongod 8.3.7 rather than recalled, and three of the five
answers were not what the corpus source assumed:
two hashed components 31303, codeName Location31303
unique on a hashed index 16764, codeName Location16764
an unknown plugin string 67, codeName CannotCreateIndex
an array at the path 16766 -- a *writeError* beside `ok: 1` on an
insert or update, and a command error from
createIndexes over data that already holds one
an array through a path refused for a *one-element* array too, which
is why `array_on_path` walks the path instead
of counting the values at it
tests/spec/indexes/hashed.json goes 0/18 -> 17/18. The one that remains
is not about hashed indexes: `find({a: null})` has to match a document
with no `a`, and this server matches only an explicit null -- with or
without an index. Next commit.
|
||
|
A.Shakhmatov
|
f151f13bfc |
db/index: partial indexes hold only what their filter selects
`partialFilterExpression` is honoured rather than refused. The filter is
consulted in exactly one place -- `build_entries` -- which is what keeps the
insert and the remove path from ever disagreeing about which documents the
index holds. Filtering at each call site instead is how an index ends up with
entries pointing at documents that are gone.
That single choke point is also why `unique` comes out right for free, and
`unique` is the whole reason this was a wrong answer rather than a missing
feature: two documents sharing a value *outside* the filter are now accepted,
where before they were E11000. Unique-within-a-subset is what the option is
for.
The filter is part of the index, so it is persisted with it: a fifth bit in
`write_index_catalog`'s flags byte and the serialized document after the TTL.
A file written before this never sets the bit and reads back exactly as it
did, so `catalog_version` stays 1 -- the same argument the free list used.
**The planner declines to read from a partial index.** It holds a subset, so
answering a query from it is only correct when the query's predicates imply
its filter, and that implication test does not exist yet. Returning too few
documents is the one failure worse than having no index at all. So it is
maintained, it enforces `unique`, and reads scan. PLAN §6.
Which predicates a filter may hold is measured: `$eq`, `$gt`, `$gte`, `$lt`,
`$lte`, `$in`, `$exists`, `$type`, `$and`, `$or`. `$ne` and `$regex` are
refused -- including a regex sent as a BSON *value*, which is how a driver
spells `{$regex: "x"}` and which the corpus caught. `sparse` and
`partialFilterExpression` may not be combined: a sparse index is a partial one
whose filter is `{<path>: {$exists: true}}`, and a document satisfying one and
not the other has no defined answer.
Same name, different filter is IndexKeySpecsConflict (86) where a differing
*option* is IndexOptionsConflict (85) next door -- measured, and the split is
that a filter decides which documents the index is over rather than how it
behaves.
ReleaseSafe earned its keep: an assertion held that a non-sparse index covers
every document after an open, and a partial one is a third shape that does
not. Extended rather than relaxed -- multikey was already the second.
partial.json 3/24 -> **24/24**. 250/250 unit tests in ReleaseFast and
ReleaseSafe, 83/83 fuzz, everything else unmoved.
|
||
|
A.Shakhmatov
|
21002528be |
commands: refuse partialFilterExpression instead of ignoring it
`createIndex({a: 1}, {partialFilterExpression: ...})` answered success, built
the index over every document, and left the option out of `listIndexes`.
An over-inclusive index still answers reads correctly -- it holds a superset,
never a subset -- so this was not the array-destroying class of bug. `unique`
is where it stopped being harmless. Measured on mongod 8.3.7:
createIndex({a: 1}, {unique: true, partialFilterExpression: {t: true}})
insertMany([{a: 1, t: false}, {a: 1, t: false}])
mongod: accepted -- neither document is in the index, so neither collides
ours: E11000 duplicate key error, dup key: {a: 1}
A legal insert refused. Unique-within-a-subset -- unique email among active
accounts, unique external id among synced rows -- is the whole point of the
option, so every use of it hit this.
Refused at creation, which is the same judgement `cmd_update` already makes
about an update spec's `sort`: "ignoring the field would be the worst of the
three possible answers." The two alternatives here were to keep enforcing
`unique` over the wrong set, or to echo the option back from `listIndexes`
while not honouring it, which is a larger lie than saying no.
CannotCreateIndex (67), joining the five existing rows of the same test with
the mutation that reddens them written beside it. The implementation is the
rest of M3's last row and comes next, against a corpus that does not exist
yet -- nothing in this repository covered the row above, in any suite.
249/249 unit tests, pinned crud corpus unmoved at 228/63/196.
|
||
|
A.Shakhmatov
|
003c418a0e |
commands: an update may be written as a pipeline
`u` (and `findAndModify`'s `update`) may be an array of aggregation stages
instead of a document of operators. The stages themselves already existed --
M2 built `compile_rewrite` and `apply_rewrite` for `aggregate` -- so this is
mostly about which of them are allowed here and what happens around them.
Six are: `$addFields`/`$set`, `$project`, `$unset`, `$replaceRoot`/
`$replaceWith`. Each rewrites one document into one document, which is the
property that makes them usable: `$match` could drop it, `$group` and
`$unwind` could change how many there are, and `$sort` means nothing to one.
mongod refuses those four by name with InvalidOptions (72) and a name that is
no stage at all with 40324, and the difference is worth keeping -- "not here"
and "not at all" are different things to be told.
`$replaceWith` did not exist here at all and now does, in `compile_rewrite`,
so `aggregate` gets it too: it is `$replaceRoot` with the expression in place
of the `{newRoot: ...}` wrapper, one stage under two spellings.
**The `_id` survives every stage.** `$replaceRoot: {newRoot: "$t"}` drops it
and `$project: {_id: 0}` asks to, and it comes back either way, because a
pipeline update rewrites a document rather than replacing one document with
another. A stage that sets it to a *different* value is ImmutableField (66);
restating the same one is fine. Measured on all six stages, and the mutation
that reddens the test is deleting the restore -- the document becomes
unfindable by the id it is stored under.
`EvalCtx.coll` becomes optional, which is what lets the upsert path run a
pipeline over a document held in memory before the collection it will be
inserted into exists. Only an `.offsets` stream reads the slab; a `.docs` one
never touched the field.
`arrayFilters` beside a pipeline is FailedToParse (9), not ignored: an
identifier a pipeline cannot spell would make the update silently different
from the one the client wrote.
pipeline.json 0/23 -> 23/23, operator corpus 125/125.
Pinned crud scorecard **218 -> 228 pass, 73 -> 63 fail**: the five pipeline
files, the four `-rawdata` files that use pipelines, and
`findOneAndUpdate-comment`, which is two more pipeline cases.
249/249 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz.
|
||
|
A.Shakhmatov
|
71e3879ae0 |
update: an unknown modifier, and two paths that decide one field
Two refusals this server did not have, both measured.
**ConflictingUpdateOperators (40).** Two paths in one update where either is a
prefix of the other at a segment boundary leave the result depending on which
operator ran first, so mongod refuses rather than picking an order:
`{$set: {a: 2}, $inc: {a: 1}}`, `{$min: {a: 2}, $max: {a: 9}}`,
`{$set: {a: 2, "a.b": 3}}`, and `$rename` counts both its ends. Siblings are
fine, and the segment boundary is what makes them fine -- `a.b` and `a.bb`
share a string prefix and decide nothing about each other, which is the
mutation the test names.
The check is quadratic in the number of paths and bounded at 64, so it stays
on the stack. Past the bound it stops checking rather than refusing: missing
a conflict in a 65-field update is a better answer than refusing a legal one,
and no driver writes updates that wide.
**Unknown modifier (9).** A name not in the operator table was
`InvalidUpdate` -> BadValue; mongod answers FailedToParse and names it. What
matters is the option nobody took: skipping an unknown operator would make a
typo an update the client believes ran. A known operator handed something
that is not a document of fields is the same code.
One existing test changed answer and was wrong: it packed
`$set: {new: 5}` and `$rename: {new: "renamed"}` into one update and passed
only because there was no conflict check. mongod refuses that with 40 --
measured, then the test split into two updates.
operators corpus 95/102 -> **102/102**. The pinned crud corpus, the positional
corpus and the aggregation corpus are unmoved at 218/73, 51/0 and 70/0.
247/247 unit tests.
|
||
|
A.Shakhmatov
|
c9575eec0a |
commands: an upsert reports the _id it generated
`Engine.insert` generates an `_id` into the bytes it writes and leaves the caller's tree without one, so every handler that looks afterwards found nothing: `update`'s `upserted` array carried `_id: null`, which the driver surfaces as `upsertedId: null`, and `findAndModify` with `returnDocument: after` returned an upserted document with no `_id` at all. The client has never seen this document. The `_id` in the reply is the only way it can name it again, so both answers were wrong in the way that matters. Settled in `build_upsert_doc` rather than in the storage engine: the upsert path is what decides the identity of the document it builds, and it has to be after the operators run because `$setOnInsert` may supply the `_id` itself. At the front of the document, where MongoDB stores it and where the `_id_` index descends on it. Found by `tests/spec/operators/`, which is the first corpus here to upsert into an empty collection and then look at what came back. The pinned crud corpus does not move -- its upsert cases match `upsertedId` loosely. set-on-insert.json 7/9 -> 8/9. 245/245 unit tests. |
||
|
A.Shakhmatov
|
1b6753fcec |
update: $setOnInsert and $currentDate
`$setOnInsert` is `$set` on the branch that inserts and nothing at all on the
branch that updates, so `Options` grows the one bit that says which -- and
`build_upsert_doc` is the only caller that sets it. It does not go through
`op_set` because of the measured exception: writing `_id` is allowed on a
document being built and refused on one being rewritten. A document nobody
has yet has no identity to change.
`$currentDate` reads a clock, and the clock is a parameter. There is no
fallback to a global one: this file has no `io` to reach the real clock
through, and the handlers pass the same `std.Io.Timestamp` that `ttl_sweep`
and the cursor sweep already read. A caller that forgets gets the epoch --
deterministic and obviously wrong -- rather than something that changes
between runs.
Measured: `$currentDate: {d: false}` writes a date. The boolean says "a date",
not "whether", and either value means the same thing. `{$type: "timestamp"}`
writes a BSON timestamp, seconds in the high 32 bits and an ordinal in the
low ones; mongod fills the ordinal from the oplog, a standalone has none, so
it is 1.
current-date.json 3/11 -> 11/11, set-on-insert.json 0/9 -> 7/9. The two left
are not `$setOnInsert`'s: one is ConflictingUpdateOperators, and the other is
a pre-existing bug the corpus found -- an upsert never reports the `_id` it
generated, because `Engine.insert` writes it into the bytes and not back into
the caller's tree. `updateOne(..., {upsert: true}).upsertedId` is null here
and an ObjectId on mongod. Next commit.
244/244 unit tests.
|
||
|
A.Shakhmatov
|
482ddb3d1a |
update: $push's modifiers are modifiers
`$slice`, `$position` and `$sort` were parsed, accepted and dropped.
`{$each: [3, 4], $slice: -3}` appended both values, sliced nothing and
answered ok: 1 with modifiedCount: 1 -- the same class of wrong answer the
positional operators were, and the reason `tests/spec/operators/` exists
rather than a list of TODOs.
Measured, and the order is the whole of it: insert at `$position`, then
`$sort` the array *including* the new elements, then `$slice` the result.
- `$slice: n` keeps the first n, `$slice: -n` the **last** n. That half is
what a capped log depends on and is the one easy to write backwards; the
test mutates exactly it.
- `$position` counts back from the end when negative, and clamps at the
front rather than wrapping.
- `$sort: 1` orders whole elements in BSON order; `$sort: {a: 1}` orders on
a field of them, and an element without it sorts as null -- the rank a
missing field has everywhere else here.
- **without `$each` there are no modifiers at all**: `{$push: {t: {$slice:
1}}}` pushes the document `{$slice: 1}` as a value. That is what makes
`$each` the flag rather than a member of the set, and it is measured, not
reasoned.
An unknown `$`-prefixed key beside `$each`, or a `$slice`/`$position` that is
not a number, is BadValue -- refused rather than ignored, which is the point.
push-modifiers.json 6/21 -> 21/21. 240/240 unit tests.
|
||
|
A.Shakhmatov
|
066b32617e |
update: $addToSet, $pop and $pullAll
`$addToSet`'s identity is `bson.compare` equality, which is already exactly
mongod's: an int32 `2` and a double `2.0` are one value, and `{a: 1, b: 2}`
and `{b: 2, a: 1}` are two, because `compare_docs` walks the pairs
positionally and tie-breaks on the key. Candidates are checked against the
array as it grows, so a `$each` holding the same value twice adds it once.
`$pullAll` is `$pull`'s neighbour and its opposite: `$pull` takes a predicate,
`$pullAll` takes values compared whole. `{$pull: {t: {a: 1}}}` removes
elements *having* `a: 1`; `{$pullAll: {t: [{a: 1}]}}` removes elements that
*are* `{a: 1}`. Sharing the comparison would have been the natural mistake and
is what the test mutates to check.
`$pop` is the small one, and its refusals are the measured part: an empty
array and an absent field are no-ops, an argument that is not 1 or -1 is
FailedToParse (9), and a non-array field is TypeMismatch (14) -- where the
identical mistake under `$addToSet` and `$pullAll` is BadValue (2). Three
codes for one shape, none of them derivable from the others. `$each` that is
not an array is 14 under `$addToSet` and 2 under `$push`, measured on both.
array-ops.json 2/26 -> 26/26. 234/234 unit tests.
|
||
|
A.Shakhmatov
|
c2f682717b |
update: $mul, $min and $max
Three operators, two shapes. `$mul` joins `$inc` in `op_arith` because they
differ only in the operation; `$min` and `$max` are not numeric operators at
all and get their own.
Measured, and each row is a rule that would have been guessed wrong:
- `{$mul: {gone: 5}}` writes **0**, not 5. An absent field starts from zero
under both operators, which is the identity for one and the annihilator
for the other, and mongod picks zero for both.
- `$min`/`$max` compare in BSON canonical order, so `{$min: {s: 5}}` on
`s: "b"` writes 5 -- a number ranks below a string -- and `{$max: {a: 1}}`
on `a: null` writes 1. An absent field is always written: there is nothing
to be smaller or larger than.
- an int32 product that does not fit widens to int64, the same ladder
`numeric_add` already climbed.
`$inc` changes answer with them: a non-numeric field or operand was
`InvalidUpdate` -> BadValue (2), and mongod answers TypeMismatch (14) with a
different sentence for each side. So two errors rather than one, and `$inc`
gets the codes it should always have had.
numeric.json 0/21 -> 20/21. The one left is `$min` and `$max` on the same
field, which is ConflictingUpdateOperators (40) -- a whole error class this
server does not have yet, and its own commit.
228/228 unit tests.
|
||
|
A.Shakhmatov
|
1482df891b |
commands/tests: name the mutation that actually reddens the arrayFilters test
The comment claimed moving `update.validate` below `scan_matching` would redden it. Ran the mutation: it does not -- the call still sits above the zero-match branch, so it still runs. What reddens it is deleting the standalone call and leaving the check to `apply`, which runs once per matched document and so never at all when nothing matched. A mutation note that has not been run is worth less than no note. |
||
|
A.Shakhmatov
|
200228b0cb |
commands: arrayFilters and the query reach the update
The engine landed last commit with nothing feeding it: `$[<identifier>]` had no filters to bind and `$` had no query to resolve against, so both refused correctly and uselessly. This is the plumbing. `arrayFilters` is read per update statement on `update` and once on `findAndModify`, which is where each command carries it. Only the *shape* is checked here -- an array, of documents, TypeMismatch (14) with mongod's own field names for either -- because which identifier a filter names, whether it is spelled legally and whether the update ever uses it all need the update's paths, and those belong to `update.validate`. `validate` is called before `scan_matching`, not inside the per-document loop, and that placement is load-bearing in both directions: an array filter the update never uses is refused (9) even when the query matches nothing at all, and an identifier nothing binds is refused (2) before a single document is read. Both measured; a test pins each, with the mutation that reddens it named in the comment. Positional corpus 23 -> 51 of 51, the gate green. Pinned crud scorecard 204 -> 218 pass, 87 -> 73 fail: the whole arrayFilters cluster, which until two commits ago was answering ok: 1 having replaced the array with a document keyed by the path segment's literal text. 222/222 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz. |
||
|
A.Shakhmatov
|
a05874d597 |
update: a positional path resolves to the elements it names
`$[]`, `$[<identifier>]` and `$` stop being refused and start being walked.
A path with a positional segment names a *set* of concrete paths rather than
one -- `y.$[].b` on a two-element array names `y.0.b` and `y.1.b` -- so
`resolve` expands it against the document at hand and every operator then
walks paths it already knew how to walk. Nothing below `resolve` knows a
positional segment exists, which is why `$set`, `$inc`, `$unset`, `$push` and
`$pull` all get it at once.
The static half -- which identifier binds to which array filter, whether a
segment may sit in first position, whether a filter went unused -- depends
only on the update and the filters, so it runs once in `validate` before any
document is touched. The document-dependent half is resolution itself: an
absent or non-array path is a refusal there, because array updates address
what is there rather than creating it, unlike `$set` on a plain path.
Codes and messages measured on mongod 8.3.7, not recalled. Sixteen shapes
were run; the ones that changed what this commit does:
- `$[]` in first position answers the *array filter identifier* message,
not the `$` one -- mongod treats the two spellings as one check.
- an array filter may have several top-level fields as long as they all
name the same identifier: `{i.b: 3, i.c: 1}` is legal, `{i.b: 3, j.b: 1}`
is not. So the check is on the name, not on the count.
- an identifier is `[a-z][a-zA-Z0-9]*`: `aB2` yes, `Ab`, `a_b`, `1x` no.
- `$rename` refuses a positional path on *either* end, with its own message
for each, so it keeps the plain split rather than resolving.
- `$unset` of an element leaves a null in its place rather than shortening
the array -- the same answer `$unset: {"y.0": ""}` already gave.
- a scalar element with more path below it is PathNotViable (28), not a
field to create. Without that check the walk would hand `y.1.b` to
`set_path` and it would replace the `7` at `y.1` with `{b: 9}` -- a
smaller copy of the destruction this whole walk replaced.
One divergence, measured and deliberate: `{"y.b": 3, "y.c": 2}` matches
`[{b: 3, c: 1}, {b: 1, c: 2}]` without either element satisfying both, and
`$` then picks element 1 on mongod and element 0 here. mongod's answer is an
artefact of which predicate last wrote its match position; guessing at it
would be worse than recording it. PLAN §6.
The command handlers still pass neither the query nor any array filters, so
over the wire this is `$[]` working and the other two refusing -- for the
right reason and with the right code, which they did not before. The plumbing
is the next commit.
221/221 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz.
Positional corpus 15 -> 23 of 51.
|
||
|
A.Shakhmatov
|
3c2ac38fd9 |
commands: drop does not unlock the collection it just freed
A use-after-free. Dispatch held `drop`'s collection lock across the handler,
`drop_collection` freed the Collection the lock lives in, and dispatch then
ran `unlock_collection` on freed memory -- an atomic read-modify-write
inside `Io.RwLock.unlock`. One insert and one drop was enough.
Two things kept it hidden for this long. `drop` had no unit test at all:
before this commit every `parse_fake_msg("drop", ...)` in the tree was in a
test written to hunt it. And over the wire it does not fault -- 25
insert/drop cycles against a live server pass -- because the general
allocator leaves the freed page mapped and the atomic write lands somewhere
harmless. That was luck, not safety: the same undefined behaviour either
way, and testing.allocator is what makes it visible, which is why the
regression test is a unit test rather than an e2e script.
Fixed by giving `drop` no collection lock at all. The catalog lock is what
actually excludes here: every collection lock in this engine -- dispatch,
the TTL sweep, `compact`'s rebuild, `write_catalog`, `slab_stats`,
reclamation -- is taken while holding the catalog at least shared, so
holding it exclusively already keeps every one of them out. The collection
lock was buying exclusion that was already there and paying for it by
locking an object about to cease existing.
That exposed a second bug rather than creating one. The dispatch epilogue --
commit, then maybe checkpoint, then maybe compact -- fired on
`locks.coll == .exclusive` as a stand-in for "this was a write". It is now
keyed on `kind == .write`, because the two agreed only by accident:
`dropDatabase` is the one write that never held a collection lock, so it has
never reached that epilogue, and a dropped database waited for some later
write to trigger the checkpoint that records it.
Mutation-checked: restoring `.coll = .exclusive` on the drop row reproduces
the original SIGSEGV in the new test.
208/208 unit (2 new) in ReleaseFast and ReleaseSafe, 83/83 fuzz, crud
scorecard unchanged at 204/87, aggregation corpus 70/0, full e2e matrix,
crash-fuzz and the 25-cycle wire drop probe green.
Left open and recorded in PLAN §6: `drop_collection` still writes no log
record, so a dropped collection resurrects on reopen unless a checkpoint ran
(pre-existing, with its own test at db.zig:5542); and `apply_pending_write`
drops under `engine.rwlock` rather than the catalog lock, so the two drop
paths disagree about which lock protects a namespace.
|
||
|
A.Shakhmatov
|
f04e7125c9 |
update: refuse a positional path instead of destroying the array
`{$set: {"y.$[i].b": 2}}` did not fail to update the array. It replaced
`y: [{b: 3}, {b: 1}]` with `y: {"$[i]": {"b": 2}}` -- every element
discarded -- and answered ok: 1, modifiedCount: 1. Remotely reachable by any
client issuing an ordinary MongoDB update.
`arrayFilters` was not implicated: the string appears nowhere in src/, the
option is accepted off the wire and dropped. The destruction was in the
path, at `set_path`'s "treat as non-array: replace with a doc" branch, so it
fired for all three spellings of "descend into this array" -- `$`, `$[]` and
`$[<ident>]` -- under every operator. `$inc` through `$[i]` stored its
operand rather than incrementing.
Two refusals, because the branch held two different mistakes:
- a positional segment is refused up front, before anything is applied, so
an update naming a good path and a positional one lands neither. Its
code is BadValue (2), which is what mongod answers for every positional
path failure.
- a plain non-numeric segment under an array -- `y.nope.b`, `y.$x.b` -- is
PathNotViable (28), measured. It is never a field to create, which is
the opposite of what `set_path` does for a missing *document* field and
the reason this branch existed at all.
Numeric segments are untouched, including the null padding past the end,
which a test now pins.
Messages are this server's own words. mongod's PathNotViable text embeds a
shell-syntax rendering of the offending element (`Cannot create field 'nope'
in element {y: [ { b: 3 }, { b: 1 } ]}`) and no BSON formatter here produces
it. The code is what the corpus asserts and the code is exact; a half-copy
of the text would be worse than a clear sentence that does not pretend.
Re-running the 17-case probe against both servers: 16 of 17 diverged before,
0 are destructive now, and 5 agree with mongod's code exactly -- every case
where mongod also refuses. The rest refuse where mongod succeeds, which is
the honest not-implemented state and is what the design review chose.
Scorecard unchanged at 204 pass / 87 fail: the 14 arrayFilters cases still
fail, now reporting the refusal rather than a corrupted document. That was
the gate this review picked -- `docs/M3_ARRAYFILTERS_DESIGN_REVIEW.md` §5,
option D -- because the corpus has no `$[]` case and no bare `$` case at
all, so passing it would have certified two live ways to destroy an array.
206/206 unit (8 new) in ReleaseFast and ReleaseSafe, 83/83 fuzz, aggregation
corpus 70/0, full e2e matrix and crash-fuzz green. Both refusals are
mutation-checked: dropping the up-front scan reddens four tests on the
array's contents rather than on the error, and restoring the destructive
branch reddens the PathNotViable test.
|
||
|
A.Shakhmatov
|
1f141ef619 |
commands: distinct
A whole command that did not exist: five corpus cases answered "no such
command". Two things about it were measured against mongod 8.3.7 rather
than recalled, and the first is not what anyone would guess.
- The answer is **sorted in canonical BSON order**, not in the order the
values were met. `{s: "b"}, {s: "a"}, {s: null}` answers
`[null, "a", "b"]`. Insertion order is the obvious implementation, it
passes every test anybody would think to write by hand against
`[11, 22, 33]`, and it is wrong.
- Deduping is the same comparator, so an int32 `1` and a double `1.0`
collapse while `null` and `"1"` survive.
Both fall out of `bson.compare`, which `$sort` and `$min` already use --
and that is not luck: mongod accumulates into a `BSONElementSet` ordered by
the same `woCompare`. The rest reuses the shared read path: byte-walked
`collect_values_bytes` for the key, so the traversal, the multikey descent
and the numeric path segments are the ones the matcher and the index
already agree on.
Also measured: a terminal array contributes its elements exactly one level
deep (`[[7, 8], 9]` gives `[7, 8]` and `9`, never 7 and 8); a missing field
contributes nothing where an explicit null contributes null; an absent
collection, an absent database and an empty key are each `ok: 1` with an
empty array rather than an error; `query` absent and `query: null` are both
an empty filter; a missing `key` is IDLFailedToParse (40414) while a
wrong-typed one is TypeMismatch (14).
Running the identical probe against both servers now agrees on every
semantic row. Three divergences remain, all outside this command and
recorded in PLAN §6: an unknown query operator matches nothing instead of
erroring (shared with find/count/aggregate, and the same class as M2's six
silent wrong answers), a non-string collection name is refused by dispatch
as BadValue where mongod says InvalidNamespace, and an unknown top-level
field is tolerated -- deliberately, since `comment` and `rawData` arrive
through that door and the corpus requires both be ignored.
crud scorecard: 201 pass / 90 fail -> 204 / 87. distinct.json 0/2 -> 2/0,
distinct-rawdata 0/1 -> 1/0. distinct-comment nets zero: its "no such
command" is replaced by the pre-4.4.14 document-comment case, which
`estimatedDocumentCount` already carries as a standing failure -- and
emulating a bug fixed in 4.4.14 for one command would make the two
disagree. distinct-collation still needs M8, but now fails with the honest
"expected 1 elements, got 2".
198/198 unit (7 new) in ReleaseFast and ReleaseSafe, 83/83 fuzz, aggregation
corpus 70/0, full e2e matrix and crash-fuzz green.
|
||
|
A.Shakhmatov
|
fc611a2c64 |
commands: $project computes, renames and narrows
The last three cases of the corpus, which is now 70 pass / 0 fail -- every
answer byte-identical to mongod 8.3.7 across the accumulators, the expressions
and the document stages.
The fix turned out to need nothing from `query.project`, which `find` shares
and which I had expected to have to rewrite. A nested spec *is* a dotted path:
`{n: {x: 1}}` and `{"n.x": 1}` are the same projection, and dotted paths are
something the existing projection already narrows correctly. So `$project` is
flattened into inclusion/exclusion flags plus a list of computed fields, and
both halves reuse machinery that was already there -- `query.project` for the
flags, `set_path` from the document stages for the computed fields. A bare path
(`{value: "$a"}`) is a rename, which is a computed field like any other.
Both shapes used to read as *falsy*, which flipped the whole projection into
its exclusion branch and returned the entire document minus the field. That was
recorded during M2 as broken rather than unimplemented; this is the fix it was
waiting for.
One case `query.project` genuinely cannot express, so it is built directly: a
projection that only computes keeps `_id` and nothing else, and with no non-`_id`
flag that function reads the spec as an exclusion and returns everything. It
cost two failures and a `id_only` flag to find, which is what a recorded corpus
is for -- the answer is obvious once seen and not before.
`$project` now goes through the same `Rewrite` path as `$addFields`, `$unset`,
`$replaceRoot` and `$unwind`, so its own branch is gone. Its refusal shrank to
the one shape mongod also refuses, mixing inclusion with exclusion, judged on
the flattened flags so a nested spec is treated like a dotted one.
191/191 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz, e2e 49, e2e2
concurrent, e2e3 16, e2e4 17, e2e6 72, e2e7 86, crud corpus unchanged at
201/90/196.
|
||
|
A.Shakhmatov
|
1a5386ff00 |
commands: the stages that rewrite a document
`$addFields`, `$set`, `$unset`, `$replaceRoot` and `$unwind`. The corpus goes
0 pass / 24 fail to 21 / 3, and the three left are `$project`'s computed
fields, renames and nested inclusions, which want the `query.project` that
`find` shares and are their own change.
**The design review was wrong about what this needed, and the corpus is what
settled it.** Tier 2 was scoped as a per-stage iterator on the grounds that
`$unwind` is 1->N and "there is no way to express that in a window over the
input". That was true of the window as it stood, and stopped being true the
moment `$project` was made to rebuild the stream instead of moving bounds over
it -- a stage that rebuilds can emit as many documents as it likes, or none. So
all five share one shape: read the window, build a new list, replace the
stream. No iterator, no rewrite.
What the recording settled, and what a hand-written test would have got wrong:
- `$addFields` whose expression resolves to nothing leaves the field out
entirely rather than setting it to null -- so `set_path` is only reached
when there is a value, and `eval_expr`'s absent/null distinction earns its
keep a second time.
- `$addFields: {"n.z": 1}` sets the nested path and keeps its siblings, and
an existing field is replaced *where it stands*, which is what makes the
stage "add or overwrite" rather than "append".
- `$unwind` drops a document whose field is missing or an empty array, keeps
one whose field is not an array *whole*, and numbers `includeArrayIndex`
from zero. Three separate behaviours where one guess would have covered
them all wrongly.
- `$replaceRoot` of a missing path and of a non-document are the same error,
40228.
`ReplaceRootNotDocument` joins `EvalError` rather than being reported at the
stage: it is a failure only a document can produce, which is the line that set
already draws.
191/191 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz, e2e 49, e2e3 16,
e2e4 17, e2e6 72, e2e7 86, crud corpus unchanged at 201/90/196.
|
||
|
A.Shakhmatov
|
37cfa863ee |
commands: the aggregation expression evaluator
The whole corpus is green: 46 pass, 0 fail, byte-identical to mongod 8.3.7 on
every case including all 27 expressions and the compound `_id` that was the
last accumulator failure.
Expressions are *compiled once per pipeline and evaluated per document*, and
that split is the point rather than an optimisation: it keeps the property M2's
refusals bought, which is that a pipeline that cannot be answered is refused
before a single document is read instead of half way through with part of the
work already reported. `Expr` is the compiled tree, `compile_expr` reports,
`eval_expr` cannot.
Nineteen operators: `$literal`, the five arithmetic ones, seven comparisons,
`$and`/`$or`/`$not`, `$cond` in both its forms, `$ifNull` and `$switch`. Plus
the two shapes that are not operators at all -- a compound document, which is
what a `$group` `_id` usually is, and an array.
Everything the corpus recorded, and none of it guessable:
- absent and a present null are *different* internally, because `$ifNull`
treats them alike and `$push` does not. Hence `?bson.Value` throughout,
where the obvious shortcut is to fold absent into `.null` at the boundary
and lose the distinction for good.
- arithmetic over absent or null is `null` -- not an error, not zero -- and
over a string is an error, 7157723.
- `$divide` by zero is 4848401, `$switch` with no branch and no default is
40069, and both are failures only a document can produce, so `EvalError`
exists and `report_eval_error` maps it.
- two operators in one expression document is 15983 and *not* `$group`'s
40238: mongod distinguishes an expression from an accumulator there.
- truthiness is MongoDB's, so `-5` is true and `0.0` is false.
- `$mod` follows the dividend's sign, so -5 mod 4 is -1.
- `$not` takes a bare argument as readily as a one-element array.
`compile_expr` and `compile_operator` call each other, so their error set is
written out rather than inferred -- Zig cannot infer a cycle, and the failure
mode is a "dependency loop" message that says nothing about expressions.
Three cases left the Tier 0 refusal test, because a compound `_id`, `$literal`
and a `$multiply` argument all work now. What is refused should be what is
missing, so an unknown operator and a wrong operand count took their place.
191/191 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz, e2e 49, e2e2
concurrent, e2e3 16, e2e4 17, e2e6 72, e2e7 86, crud corpus unchanged at
201/90/196.
|
||
|
A.Shakhmatov
|
76afa75efe |
commands: the $group accumulators
Nine of them -- `$sum`, `$avg`, `$min`, `$max`, `$first`, `$last`, `$push`,
`$addToSet`, `$count` -- and the corpus goes 9 pass / 10 fail to 18 pass /
1 fail. The one left is the compound `_id`, which needs the expression
evaluator.
Landed before that evaluator, against the tier order the design review set
out, and the corpus is why: every one of these takes a single value per
document, a path or a constant, so nine of its ten failures turned out to be
reachable without one. `classify_expr` already produced exactly that value.
What the recording caught, which is the argument for measuring expectations
rather than writing them:
- `$avg` over a group with no numeric value is **null**, not `0`. A divisor
that counted documents rather than numbers would pass every test anybody
would think to write by hand, and be wrong on the one group that matters.
- `$min`/`$max` compare across types in canonical BSON order, so the maximum
of `30`, `7` and `"not a number"` is the string.
- `$push` skips an absent field but would push an explicit null, so "resolved
to nothing" and "resolved to null" cannot be the same value internally --
which is why the accumulators take `?bson.Value` and not `.null`.
- `$first`/`$last` follow input order, including when the value is absent:
`$last` of a missing field is null, not the last present one.
`AccState` is one struct rather than a union: the fields are small and every
site already switches on the kind, so a union would add a tag test where a
switch was going to be anyway. Its arrays are the gpa's, the values inside them
the reply arena's -- they outlive the group and travel with the documents.
`numeric_value` is the int32-or-double narrowing MongoDB reports, shared now
between the accumulators and `cmd_aggregate`'s count fast path. It was written
twice before; a divergence between them would make `countDocuments` disagree
with the pipeline it is a shortcut for.
`$avg` and `$push` came out of the Tier 0 refusal test, replaced by
`$stdDevPop` and `$mergeObjects`. The refusal is a property of what is missing
rather than of a list, and the test should read that way.
191/191 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz, e2e 49, e2e3 16,
e2e4 17, e2e6 72, e2e7 86, crud corpus unchanged at 201/90/196.
|
||
|
A.Shakhmatov
|
8ebeb9d4ec |
commands: $out and $merge, written by the dispatch epilogue
The seven reachable failures of M2, and the first commit of the milestone to move the scorecard: 194/97/196 -> 201/90/196, with `aggregate-*.json` going 9 pass / 13 fail to 16 pass / 6 fail. The seven that moved are exactly the seven priced as reachable, and the six that remain are exactly the six attributed to M2.5 ($addFields, the expression engine), M4 ($listLocalSessions) and M8 (collation). Both stages write to a collection the pipeline is not reading, and three things stood against doing that in the handler: `aggregate` is a `.read` command, dispatch takes locks from a static table keyed on the command name before the handler runs, and `Collection.lock` allows exactly one collection lock at a time. So the handler computes the output under the locks it has and leaves it in `Context.pending_write`; the epilogue applies it with nothing held, beside the commit and the checkpoint already there. The `.read`/`.write` contract is amended in its own comment rather than quietly broken. `pending_write` is cleared at the top of every dispatch, so a handler that errors before setting one cannot leave the previous command's write to fire. A failed write replaces the pipeline's `ok: 1` with the failure, because a client told the aggregation succeeded would believe the collection had been written. What the stages do not implement is refused, not ignored: `$merge`'s `whenMatched`, `whenNotMatched`, `on` and `let` all select behaviour this server does not have, and a `whenMatched: "fail"` that silently merged would be the same lie Tier 0 spent three commits removing. Codes measured against mongod 8.3.7. `$out` and `$merge` answer byte-identically to it on both the replace and the upsert case. NOT ATOMIC, and said out loud in the code rather than left to be discovered. mongod replaces an `$out` target atomically; this engine has no cross-collection atomicity and no rename to build one from, so a crash between the drop and the last insert leaves the target holding part of the new output where MongoDB would leave the whole of the old. The fix is write-to-temp-and-rename and rename is a command that does not exist here. The test's mutation is the argument for the epilogue in one line: apply the write inside the `$out` branch and it deadlocks rather than fails. 191/191 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz, e2e 49, e2e2 concurrent, e2e3 16, e2e4 17, e2e6 72, e2e7 86. |
||
|
A.Shakhmatov
|
61ce9805c7 |
commands: $project is a stage, not a note about how to print the answer
It set a variable that the emit applied once. Three consequences, all measured
on a live server before this changed rather than read off the code:
- only the *last* `$project` in a pipeline had any effect;
- a `$match` after one still matched on the field it had removed;
- `{y: {$literal: 5}}` read as falsy, which flipped the whole projection into
its exclusion branch and returned every document minus `y`, where mongod
adds a computed `y`.
Now it transforms the stream where it stands, materializing into the same tree
form `$group` already produced. `$match`, `$sort` and `$group` after it read
what it left, which is what "stage" means.
What it cannot do is refused rather than mis-read. A computed field needs the
expression evaluator M2.5 brings, and a nested spec (`{a: {b: 1}}`) needs a
narrowing `query.project` does not do -- both were falsy, and falsy is the
answer that quietly returned the whole document. Codes read off mongod 8.3.7,
and the replies now match it on code, codeName and message text for the empty,
the mixed and the computed forms:
projection must have at least one field 51272 Location51272
cannot mix inclusion and exclusion 31254 Location31254
unknown expression in $project 31325 Location31325
Recorded, not fixed here: `query.project` is shared with `find`'s `projection`,
where a nested inclusion has the same falsy reading and the same wrong answer.
That is a real gap rather than an unimplemented feature -- narrowing is
something this projection should do -- so it is its own fix, not a refusal, and
it belongs with whichever milestone takes `find`'s projection seriously.
Scorecard unchanged again at 194/97/196. The corpus has no `$project` stage
tests either; A6 said so, and this is the second commit to confirm it.
Mutations: drop the refusals and the empty projection answers `ok: 1`; put the
projection back on the emit and the `$match`-after-`$project` case goes red.
190/190 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz, e2e 49, e2e2
concurrent, e2e3 16, e2e4 17, e2e6 72, e2e7 86.
|
||
|
A.Shakhmatov
|
d9772c4ed5 |
commands: a pipeline may group what an earlier stage generated
A remote crash, present before this milestone and reachable by any client with
a two-stage pipeline:
aggregate: [{$group: {_id: "$k", n: {$sum: 1}}},
{$group: {_id: null, g: {$sum: 1}}}]
thread panic: index out of bounds: index 2, len 0
`$group` took `[]const u64` and was handed `offs.items[start..end]` whatever
the stream was made of. A pipeline starts as slab offsets -- matched and
reordered in place, never materialized -- and flips to generated documents the
moment a stage produces something the slab does not hold. After that `offs` is
empty while `start`/`end` count trees, so the slice ran off an empty list and
took the server thread down. There is no authentication in front of it.
Found while making `$project` a real stage, which reaches the same branch;
confirmed against the binary at the previous commit rather than assumed, so
this is a pre-existing defect and not a regression of that work. It is
committed on its own for that reason.
`Stream` names the two forms the rest of the pipeline had been carrying
implicitly in `in_trees`, and `$group` now reads whichever is live. The slab
side stays byte-walked -- grouping a million documents does not build a million
trees to read one field -- and `path_in_pairs` is the tree counterpart of
`query_path_value_bytes` for the other half.
The test groups by a key and then counts the groups, and sums `$n`, a path that
only resolves against the generated document. Its mutation -- hand `run_group`
the offsets unconditionally -- aborts the run rather than failing it, which is
why this wanted a test and not a code read.
Answers byte-identically to mongod 8.3.7 on the pipeline above.
189/189 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz, e2e 49, e2e3 16,
e2e4 17, e2e6 72, e2e7 86.
|
||
|
A.Shakhmatov
|
ae48e05a19 |
commands: $group refuses what it cannot compute
M2 Tier 0, and the first commit of the milestone: stop answering wrong numbers
with `ok: 1`.
`$group` had one accumulator, `$sum`, and one expression vocabulary, `$field`
paths and constants -- open-coded twice and applied to whatever arrived.
Everything outside that fell through to a zero. Measured on a live server
before this commit: `{$avg: "$x"}` answered `0`, and so did `$max` and `$push`;
a compound `_id` collapsed every document into one group keyed by the
unevaluated expression; `{$literal: 1}` came back echoed; `{$sum: {$multiply:
[...]}}` was `0`. Six of eight probed pipelines succeeded with a wrong result.
That is a worse failure than an unimplemented one. An unrecognised stage is a
bug report; an `$avg` that returns `0` is a corrupted report nobody files. It
is also invisible to the gate: the corpus this project runs has no aggregation
stage tests at all, which is what PLAN amendment A6 is about.
So the vocabulary is now named -- `GroupExpr` is a path or a constant, and
`classify_expr` is the single place the `$`-prefix distinction is made -- and
every accumulator is validated before a document is read. A pipeline that
cannot be answered is refused whole rather than half-answered.
Five error codes added, every one read off mongod 8.3.7 rather than recalled,
and the replies now match it on code and codeName for all six `$group` shapes
probed:
unknown group operator 15952 Location15952
a group specification needs _id 15955 Location15955
accumulator is not an object 40234 Location40234
two operators in one 40238 Location40238
unrecognized expression 168 InvalidPipelineOperator
`$avg` and friends are reported as *unknown* operators, which is the choice
`location_unrecognized_stage` already made for `$addFields`: this server
reports what it does not implement using MongoDB's own code for "no such
thing", because a code MongoDB never emits would break the error-code parity
every milestone is held to. The message names the construct.
`$sum` over a non-number still contributes nothing -- that is MongoDB's rule,
not a stand-in for an unimplemented one, and the comment says so where it
would otherwise read as another silent zero.
The scorecard does not move: 194/97/196 before and after, byte-identical. That
is not a disappointment, it is the design review's central finding arriving on
schedule -- the corpus cannot see any of this, which is why M2.5 has to bring
its own.
Mutations: drop the accumulator-name check and `$avg` answers `ok: 1` again;
drop the `_id` classification and the compound `_id` does.
188/188 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz, e2e 49, e2e2
concurrent, e2e3 16, e2e4 17, e2e6 72, e2e7 86.
|
||
|
A.Shakhmatov
|
a748a3d08c |
db/pager/tests: cleanup pass over the free list
No behaviour is meant to change and the gate confirms it: 1.94x / 679.2 MB
reclaimed / 9 rebuilds and 2.46x / 934.0 MB / 13 on the two 16 KiB lines,
0.0 MB on the 200-byte line, all identical to the numbers recorded for them.
Deduplication. `pages_for` was written in db.zig and again in pager.zig and
twice more inline; there is now one, public, and the two pre-existing copies
call it. `pages_per_map_align` replaces three hand-rolled `map_align /
page_size`. `SlabRun.window_first` was a stored field that could never legally
disagree with `first` and was maintained by hand at two sites -- now a method.
`keep_piece` re-derived `SlabRun.window_count` character for character; it
calls it. `insert_run` scanned linearly for a position `run_of` binary-searches
for, which made loading a fragmented catalog quadratic; both now go through one
`run_lower_bound`. Freeing a run's window map was written four times; one
helper. The 20% rebuild share was stated in `note_compact` and again in
`wants_rebuild`, with a comment arguing at length that they must be the same
number -- `worth_rewriting` makes that structural.
Efficiency. The identity assert in `reclaim_windows` called `dead_located()`,
an O(every window) walk, and `assert_msg` is live in ReleaseFast -- so it
doubled the scan the reclamation was about to make (2.75 MB streamed twice per
reclaiming checkpoint at the 21 GB the gate targets). `dead_located` is now a
maintained counter, the check is O(1) in every build, and the scan cross-checks
it while it is there. `SlabRun.full` lets a run with nothing to give be copied
without its counters being read at all, so the common case is O(runs) rather
than O(windows).
The pager's two allocation policies were hand-copying the claim step, and the
copy had already lost two of the three preconditions -- `alloc_slab_run` never
checked `pages <= reserved_pages`. Both now go through `claim_locked`.
`reclaimed_bytes` moves from Collection to Engine, beside `compactions`, which
is how it is read and the only place it can be honest: a life-of-the-process
total must not lose a dropped collection's share. Both join `Counters`, so
`slab_stats` stops opening `counter_lock` by hand.
The ownership assertion in `write_catalog` was gated on `is_test or Debug`, a
predicate nothing else in the codebase uses, which left the one silent failure
this design can produce unchecked in ReleaseSafe. It is now `!= ReleaseFast`,
the line `protect_stable` already draws. Measured: no change to the suite's
runtime.
Altitude. `note_checkpoint` was called from exactly one place, the tail of
`upsert` -- so a delete armed no checkpoint by any route, which is why
reclamation only ever ran when the *rebuild* trigger fired and the rebuild then
reset the window map it would have used. `remove` and the TTL sweep arm one
now, next to the `note_compact` calls that were added for the same omission a
milestone ago. `compact`'s leading checkpoint stays, demoted in its comment
from the mechanism to the local ordering it actually guarantees.
serverStatus reports `allocTailBytes`/`freeReadyBytes` instead of page counts,
so the harness stops hard-coding 4096 -- the kind of constant this milestone
was blindsided by once already.
tests/e2e/churn.js: `deleteMany({_id: {$in: [5000 ids]}})` exceeded
`index.max_combos`, so the planner refused the index and every delete became a
full collection scan re-filtering each document against 5000 members. That was
the entire runtime of the harness. One delete spec per id instead: the 40k x
16 KiB gate goes 57 s -> 6 s, and the 150k x 200 B line 483 s -> 3 s, with
identical output. Also: the per-round `countDocuments` is gone (the harness
knows the count), and the server log is a bounded ring rather than a rope that
grows with everything the server ever said.
Reverted from the review: reusing one MongoClient across the startup poll. A
client whose first connect fails tears its topology down and every later
command on it fails identically, so it turns "not up yet" into "never comes
up" -- it broke the first run. The reason is now a comment.
187/187 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz, e2e 49, e2e2
concurrent 2 and the crash pair, e2e3 16, e2e4 17, e2e6 72, e2e7 86,
crash-fuzz 60 cycles.
|
||
|
A.Shakhmatov
|
8bf35e707c |
commands: serverStatus reports what the slab is doing
A `multifora` subdocument -- named so nobody mistakes it for a MongoDB section -- carrying `liveBytes`, `deadBytes`, `slabBytes`, `reclaimedBytes`, `slabRuns`, `freeReadyPages`, `allocTail` and `compactions`. The milestone's gate cannot be read without them. A steady-state size ratio can look respectable while reclamation does nothing at all: the file grows, a rebuild periodically halves it, and the average comes out fine. What distinguishes the two is `reclaimedBytes` rising while `allocTail` stays put, and no ratio shows that. Same for rebuilds -- "the ratio improved" and "the ratio improved because reclamation worked rather than because a rebuild ran" are different results, so `compactions` counts collections rewritten. The byte figures are summed from the collections rather than read off the engine's running totals, so this reports the same side of the comparison `checkpoint` asserts. A counter that had drifted from the catalog would otherwise make the gate measure the drift. Every field is present at zero. A gate that cannot tell "no pages ready" from "field missing" cannot be read at all. 186/186 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz, and the full e2e matrix. |
||
|
A.Shakhmatov
|
0dd9a0c908 |
db: a rebuild copies only the collections that have garbage
`compact` walked every collection unconditionally, so garbage in one paid for a full copy of the other thirty-nine. A copy is not free even where it reclaims nothing: it rewrites every document and every index, and it bumps the layout epoch, which kills every open cursor on a collection that had no reason to be touched. The gate is the share `note_compact` already applies to the engine's totals, and using the same one is what keeps them from disagreeing. If no collection passes it then `dead_i < live_i / 4` for every one, so `sum(dead) < sum(live) / 4` and the engine's trigger could not have fired either -- a compaction that runs always rebuilds at least one collection and cannot spin re-arming itself over garbage no rebuild will take. An absolute floor per collection would break exactly that: forty collections each under the floor can sum to well over it. Two existing tests needed real garbage, which is the change working. "a rebuild kills an offsets cursor and spares a streaming one" and "the epochs that invalidate a cursor move exactly when they must" both called `compact` on a clean collection and relied on it rewriting anyway. The first now rewrites all 60 documents (a replace, not a delete, so the drain still checks that the stream yields exactly 60 once each -- and with a changed field, since an identical replace is deliberately not a write). The new test asserts both halves: the dirty collection's epoch moves, the clean one's does not, and the clean one's `slab_used` is unchanged -- a repack starts the slab over, so that number could not survive one. Mutation: delete the guard, and the clean collection's epoch moves too. 185/185 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz, e2e 49, e2e2 concurrent 2 and the crash pair, e2e3 16, e2e4 17, e2e6 72, e2e7 86, crash-fuzz 60 cycles. |
||
|
A.Shakhmatov
|
3e2d0ab134 |
commands: a count reply is an int32
Two of the tests added alongside endSessions read the count reply's `n` as int64. ReleaseFast does not check the union tag, so both passed there and aborted in ReleaseSafe -- which is the whole argument for running the suite in both, and the reason the plan makes ReleaseSafe non-optional. 175/175 in ReleaseFast and ReleaseSafe. |
||
|
A.Shakhmatov
|
f5c827e581 |
commands: endSessions judges the array it is handed
It stays a no-op -- there is nothing to end -- but stops being a blind `ok`. This is the one session command that arrives in normal operation: a driver sends it on close for every session it handed out. Validating an argument that is then discarded looks like ceremony, and is not: `ok: 1` to a malformed `endSessions` says the server understood something it never read, which is the same class of answer the previous commit removed for `txnNumber`. Codes and messages measured against mongod 8.3.7, including the field path `endSessions.endSessionsFromClient` -- an IDL artefact, since the command's own field is `endSessions` and the parsed argument carries a different name. It is reproduced rather than tidied: a path that differs from the real server's is worse than an odd one. With this the raw probe's replies are identical to mongod's for every lsid and endSessions shape. What is left is five deliberate divergences, all recorded: three because mongod knows per command whether `txnNumber` is even accepted and answers Location50889 or 263 before reaching the standalone refusal, and two because `startSession` and `refreshSessions` are not implemented -- a driver calls neither, generating session ids locally, so CommandNotFound is the honest answer. `commitTransaction` joins them, and it can only be reached by a client whose write this server already refused. 175/175 unit tests. |
||
|
A.Shakhmatov
|
54092f44be |
commands: accept a well-formed lsid, refuse what would be a lie
A driver puts `lsid` on every acknowledged command already, because `add_server_info` advertises `logicalSessionTimeoutMinutes`. So this is not new plumbing, it is a decision about input that has been arriving all along and being ignored. Accepting it and doing nothing is honest: a session here would own nothing -- no transactions to scope, no retryable writes, cursors that outlive their connection for their own reasons. There is deliberately no session registry; it would be a mutex on the dispatch path guarding state nothing reads, and M4's transaction state machine is what should decide its shape. `txnNumber` is a different matter, and ignoring it would be the lie this commit exists to remove. A transactional write would run non-transactionally, answer `ok`, and become durable; the client would find out at `commitTransaction`, by which time the data is on disk. It is refused, with `startTransaction` and `autocommit` alongside it for the same reason. Every code and every message was measured against mongod 8.3.7 through a raw OP_MSG probe -- the driver overwrites `lsid` with its own session, so a malformed one cannot be sent through it and none of this was checkable the usual way. Three things the measurement settled that guessing would have got wrong: an unknown command with a malformed lsid answers CommandNotFound, so the lookup comes first and this check belongs exactly where it sits; the codes for a bad session id are IDL parser codes (40414, 40415) rather than anything resembling the rest of our table; and a bad UUID length is InvalidUUID 207 while a bad subtype is TypeMismatch 14, which no amount of reasoning would have produced. Three divergences from mongod, all one cause: it keeps a per-command table of which commands accept `txnNumber` at all, and answers Location50889 or OperationNotSupportedInTransaction 263 for those that do not, before reaching the standalone refusal. We have no such table and give the standalone answer uniformly. For every CRUD command -- everything a driver would actually send these fields on -- the replies are identical; they differ only on things like `ping`, where mongod is more specific rather than differently right. Checked before any lock is taken, and the test for that is the second half of each refusal: keep using the engine afterwards, with a write that has to take the catalog exclusive to create a collection. Mutation-checked by moving the call below `lock_catalog` -- the test run hangs, which is precisely how the nameless-command lock leak presented, since a leaked *shared* lock is invisible to every reader. 174/174 unit tests. |
||
|
A.Shakhmatov
|
548c882d47 |
commands: the wire version says what the version string says
`buildInfo` has always reported 4.4.0 and the handshake has always reported maxWireVersion 8, which is 4.2. A driver believes the wire version: it refused client-side to send `hint` on an unacknowledged delete or findAndModify (the error is "only supported on MongoDB 4.4+", raised without a round trip), and withheld `comment` from getMore, listCollections and listDatabases (lib/operations/get_more.js:43 and its neighbours). Both are things this engine handles -- the acknowledged hint suites pass, and getMore ignores fields it does not know -- so the effect was purely the number disagreeing with itself. The 8 was not arbitrary. The comment above it tied it to omitting `topologyVersion`, which is what keeps a driver off the streaming hello protocol we do not implement -- a real bug, once visible as Compass reconnecting every heartbeat. Checked before touching it, in the driver rather than from memory: `useStreamingProtocol` (lib/sdam/monitor.js:154) returns false whenever `topologyVersion` is absent and never looks at the wire version at all. The omission is the whole mechanism; the wire version was a second line of defence that never existed. The comment now says so. A test asserts the two agree, so they cannot drift apart again silently, which is the actual defect here -- not the value. Spec suites 173/118/196 -> 189/102/196: 16 cases fixed, none broken. Ten are the unacknowledged-hint cases the previous commit uncovered, six are `comment` forwarding. 166/166 unit tests in ReleaseFast and ReleaseSafe, 82/82 fuzz, e2e 49, e2e2 concurrent 2 + crash pair, e2e3 16, e2e4 17, e2e6 72, e2e7 86, crash-fuzz 60 cycles. |
||
| f2844e7894 |
cursors: server-side cursors for find, aggregate and the listing commands
Every reply came back in a single batch with `cursor.id = 0`, `getMore` was a
stub answering an empty `nextBatch` on the literal namespace `test.$cmd`, and
nothing read `batchSize`. That caps the useful collection size at what fits in
one 48 MiB message, which is the opposite of the tens-of-GB target and the
reason M0 made whole-index scans stream: the streaming candidate generator
existed with no consumer that could suspend.
## What a cursor is allowed to remember
A cursor holds no lock between requests, so everything it saves has to survive
arbitrary concurrent mutation. Nothing here is a pointer, and the two things
that look like stable addresses are not: `reset_tree` re-creates node ids 0 and
1 as different nodes, and `rebuild_collection` moves every document. Three
sources, chosen by query shape, each with a different memory contract:
- **stream** -- an index-ordered walk resumed from a `(key, off)` anchor plus a
`(leaf, slot)` hint. O(key) state, so this is what lets a cursor walk a
collection larger than memory. Survives a rebuild, because a repack changes
no key.
- **offsets** -- the matched slab offsets a narrowed plan already materialized,
8 bytes each. Killed by a rebuild with `QueryPlanKilled`, because those
offsets now name unrelated bytes.
- **buffered** -- canonical BSON copies, for a sort no index provides and for
aggregate/listing output. Depends on nothing, which is what lets a listing
hold a cursor over a `$cmd.*` namespace no collection backs.
`Collection.layout_epoch` and `Index.epoch` are the invalidation tokens, both
checked as error returns rather than assertions since a client reaches them by
keeping a cursor open across maintenance.
## Resume
`resume_forward`/`resume_reverse` are O(1) while the hint holds and fall back to
an exact-order band walk bounded by `resume_walk_max`. Without the hint, `seek`
lands at the *start* of an equal-key band, so `sort({status: 1})` over three
distinct values across 10M documents would cost ~5e10 comparisons to drain.
Two hazards found by draining a collection while writing to it, neither
predictable from reading the code:
- A deleted anchor must resume at its *band position*, or the rest of an
equal-key band is silently dropped -- most of the collection on a
low-cardinality index. Hence `band_index`.
- On a **unique** index a same-key entry can only be the anchor rewritten, so
resuming at it returned updated documents twice. Observed as duplicate `_id`s
while updating underneath a drain.
## Protocol
Measured against mongod 8.3.7 rather than recalled, which corrected three
assumptions: a bare `getMore` does *not* inherit the find's `batchSize` (4998 of
5000 documents come back), a namespace mismatch is `Unauthorized` (13) not
`CursorNotFound`, and `CursorInUse` is 143 not 12051.
`internalQueryFindCommandBatchSize` is 101, `cursorTimeoutMillis` 600000,
`clientCursorMonitorFrequencySecs` 4.
The rule everything follows is **never look ahead**: a batch that met its target
leaves the cursor open even when the source is in fact exhausted, so four
documents at `batchSize: 2` take three commands. `limit` acts as an EOF source,
which is what makes `batchSize == limit` close in one round trip. `skip` is
consumed once. `batchSize: 0` returns an empty batch with a live cursor.
Cursor ids are `(nonce << 20) | slot`, always positive. The nonce is not
decoration: without it a recycled slot serves one client another's documents.
Cursors are not connection-pinned, since the driver spec allows a `getMore` on
any connection to the same server; they end at exhaustion, `killCursors`, or the
idle sweep (a second monitor fiber, separate from the TTL one because the
cadences differ by an order of magnitude and a TTL failure must not stop
reclamation). The registry is fixed-capacity and evicts the least recently used
cursor, whose client sees the same 43 an idle timeout gives.
Fixed alongside, because cursors are what expose them:
- `listCollections` reported `"<db>."` with an *empty* collection part, which
makes the driver throw client-side -- so it would have broken the moment its
cursor stopped being id 0. Now `<db>.$cmd.listCollections`, as mongod uses.
- `count` ignored `skip` and `limit` entirely.
- `wire.end_message` now bounds a reply by the 48 MiB we advertise rather than
by `maxInt(u32)`; a reply past what we told the client to expect is not a large
reply, it is a desynchronized connection.
- Two `codeName` strings were wrong: 72 is `InvalidOptions` (MongoDB has no
`InvalidArgument`), and 40324 reports as `Location40324`.
## Verification
Unit 160/160 in ReleaseFast and ReleaseSafe; `tests/e2e/e2e7.js` adds 86 cursor
checks across five phases (batching/lifecycle/errors, streaming across churn,
aggregate+listings+count, expiry+capacity, restart) and is self-contained
because cursor behaviour is only observable with non-default flags. No
regressions: e2e 49, e2e3 16, e2e4 17, e2e2 2, e2e6 72. Spec 168 pass / 124
fail, +5 against the previous scorecard.
Mutation-checked, per the repo's second ground rule: `hint_slot + 1`, the
`band_index` off-by-one, both epoch bumps, the id nonce, the at-least-one-
document rule, and `stream_shape` returning null each turn the intended test
red. One claim was withdrawn rather than kept -- swapping `std.mem.order` for
`cmp_prefix` in the band walk changes nothing observable, so the comment now
says so instead of asserting a check that does not hold.
|
|||
| 21489723a9 |
db: a write that changes nothing is not a write
`nModified` counted every write, so an update that altered nothing was reported
as a modification. MongoDB counts a document as modified only if applying the
update changed it, and writes no oplog entry when it did not: `$set: {x: 11}`
on a document already holding `x: 11` is matched and not modified. The spec
suite says it plainly -- `bulkWrite` with four updateOne operations expects
matchedCount 2 and modifiedCount 1.
Decided in the engine rather than the command, because that is where the
document is already serialized: the comparison is against the bytes that would
actually be stored, and it lands before the log append, so a no-op costs no log
record, no fsync, no slab bytes and no garbage. `Engine.replace` returns
`Written.modified` or `.unchanged` and `cmd_update` counts the first.
That exposed a second difference. A replacement keeps `_id` at the front, so
replacing a document with itself was a byte-level change whenever `_id` was not
stored first -- and it usually was not: the Node driver fills a missing `_id` by
assigning the property, which in JavaScript appends it, so `insertOne({name,
age})` reaches the server as `{name, age, _id}` and we stored it that way.
MongoDB moves `_id` to the front whatever order it arrives in. Now so does
`serialize_with_id`, for every document rather than only the ones whose `_id` it
generates. Visible to clients as `_id` coming back first, as it does from
MongoDB.
spec scorecard 161 pass / 131 fail -> 163 pass / 129 fail
bulkWrite.json 8 pass / 2 fail -> 10 pass / 0 fail
e2e.js 45 checks -> 49
No spec file regressed. Mutation: delete the byte comparison in `upsert`'s
`.replace` arm -- red on the log growing, on the garbage counters moving, and on
`replace` claiming `.modified`.
|
|||
| 53f88e6d3b |
update: replacement-style writes
`replaceOne`, `findOneAndReplace` and `bulkWrite`'s `replaceOne` all failed
with "bad update". `update.apply` rejected any update document whose first key
was not `$`-prefixed, so a replacement document -- which by definition has no
operators -- could not get through at all.
MongoDB decides on the first field and nothing else: `$`-prefixed means
operators, anything else means the document *is* the new content. An empty
document is a replacement too, and a legal one. `is_replacement` says which,
`apply_replacement` does the work, and because all three call sites already go
through `apply`, that one branch covers the update command, findAndModify and
the upsert builder.
What a replacement means, precisely:
- every field is replaced except `_id`, which is immutable and keeps its
position at the front, where it is stored and where the `_id_` index
descends on it;
- a replacement may restate the same `_id` but not a different one -- that
is `ImmutableId`, because otherwise a rewrite would silently change a
document's identity while the index entry kept the old key;
- when the target has no `_id` yet, the replacement supplies it. That is the
upsert path: `build_upsert_doc` seeds a document from the filter's
equalities, so `replaceOne({_id: 99}, {u: 1}, {upsert: true})` inserts
`{_id: 99, u: 1}` and not a generated ObjectId;
- a mixed document is refused from either side, rather than guessed at.
Two options on update specs are refused rather than ignored:
- `multi` with a replacement (FailedToParse). A replacement describes one
document; applying it to many would leave every match identical apart from
its `_id`.
- `sort`, a MongoDB 8.0 addition this server does not implement. Ignoring it
is the worst of the three answers -- `sort` chooses *which* match to write,
so the client would silently get a different document than it asked for.
spec scorecard 131 pass / 161 fail -> 161 pass / 131 fail
e2e.js 35 checks -> 45
Sixteen spec files improved and none regressed. The two `-sort` files briefly
did: they had been passing on their "server-side error" case, which our
"bad update" failure satisfied by accident, and passing for the wrong reason is
how a gap survives a scorecard.
Five mutations, each verified red: seeding the replacement from the old pairs,
dropping the `_id` comparison, dropping the `_id` a replacement supplies, and
removing the mixed-document guard from either loop.
Note `nModified` is still wrong for a write that changes nothing -- MongoDB
counts a document as modified only if applying the update altered it. That is
the remaining bulkWrite failure and is fixed next, separately.
|
|||
| 138b7f706f |
db/storage: reclaim the log once a checkpoint covers it
The point of a lagging checkpoint: a record whose effect the data file already holds is redundant, so the log can go back to just its header. Without this the log only grows and every open pays for every write ever made. Ordering, which is the whole safety argument: publish the watermark, *then* truncate. The other way round, a crash between them leaves the records gone from the log and absent from any image. A failed truncation is a warning rather than an error -- it costs space and replay time, and loses nothing, so it must not fail a checkpoint that already succeeded. Also wires checkpointing up, which nothing did before. `note_checkpoint` arms it when the log passes a threshold, and the write epilogue and the TTL monitor both claim it -- outside any collection lock, for the same reason compaction runs there: it takes the log lock. The threshold is separate from the compaction one on purpose: compaction is about the garbage share of the data, a checkpoint is about how much replay an open would otherwise do. -- Two things the tests taught me. The first version measured the log before the checkpoint and found 16 bytes -- just the header. Appends buffer in the log's open block and only a commit seals and writes it, so there was nothing on disk to shrink. The test commits first now, and says why. And the "no valid watermark" warning fired for every young database, which is its normal state before the first checkpoint. It now distinguishes a watermark that was *written and cannot be read* from one that was never written -- warning about the ordinary case is how people learn to ignore the warning that matters. Mutation-checked, red: skipping the truncation. Not covered, and the test says so: moving the truncation before the publish, whose failure mode is a crash landing between the two. That needs process-level crash injection, which an in-process test cannot express. |
|||
| 9390021b1e |
index/commands: stream whole-index scans; add a reverse leaf iterator
A whole-index read used to materialize every candidate before the caller saw
the first one. At the tens-of-GB target that is a list of every offset in the
collection -- ~160 MB for a countDocuments({}) over 20 million documents --
which defeats the point of moving storage to disk. Cursors are M1, but
*streaming a scan* has to exist now.
`Candidates` is the one loop candidates arrive through, whatever produced them:
a plan's materialized lookups, or the index read end to end. That keeps this
file's governing invariant -- an index only generates candidates, the full
filter is re-applied to every one -- in a single place. A narrowed plan still
materializes, because its multikey/$in dedupe genuinely needs the whole set and
is bounded by selectivity.
`RevIter` walks `Node.prev`, which has always been maintained and which nothing
had ever read: a descending scan materialized the whole index and reversed the
list. `find({}).sort({_id:-1}).limit(20)` becomes O(20).
The unfiltered fallback now walks the _id_ index instead of the docs map. That
is ordered rather than hash-ordered, and it does not depend on a structure that
is about to be deleted.
`Plan.full_scan()` refuses multikey indexes, since one document contributes
several entries there and a stream cannot dedupe what `search` did. The check is
currently redundant -- the planner refuses to order a multikey index anyway --
and is kept because the two guards protect different things. Stated precisely
in both places after checking: the commands.zig test reddens only when *both*
guards are removed, which is what that test actually pins.
--
This also broke e2e6's compaction check, and the fix there is the more
interesting half.
The check required peak/final > 1.4 and got 1.28. The final size was identical
to the byte (2,398,065 vs 2,398,064) -- compaction reclaimed exactly as before
-- and only the peak moved. Isolated to one variable: changing just the order
updateMany({}) walks its matches moves peak/final between 1.65 and 1.28, because
compaction can also fire from the once-per-second TTL monitor and whether one
lands inside the batch shifts the peak a long way while leaving the outcome
unchanged. The threshold was measuring the schedule.
Replaced with `peak > final`, which measures the shape instead: an append-only
log grows monotonically, so its maximum *is* its final size, and a file that
was ever larger than it ended can only have been rewritten.
Worth recording why the obvious alternative does not work. An absolute size
bound cannot distinguish a working compactor here: the payload is one repeated
character, so ~48 MB of records LZ4-compress to ~3 MB whether or not anything is
reclaimed -- with compaction disabled entirely the file still ends at 3.1 MB. I
first wrote the comment claiming that bound was the strong one, then measured it
and found the opposite; `peak > final` is what goes red.
|
|||
| 491a4d0a6a |
index: a leaf record's payload becomes the document's slab offset
PLAN amendment A3. The B+tree leaf had nowhere to put a document's slab offset -- `Slot.extra` is the payload length for a leaf and the child node id for an internal separator -- which is what blocks the `_id_` tree from becoming the primary lookup once the docs hashmap goes away. A leaf record is now `key ++ offset_le`, so `extra` is always 8 and every byte-accounting site (fits, record_cost, slot_cost, balanced_cut, repack_keep_prefix) is untouched. Records get *smaller*: an ObjectId `_id_` record goes from 26 bytes to 21. `Entry.id` is deleted rather than re-owned. Every entry one document contributes shares one document, so which document it is belongs on the call that commits the entries -- which also makes it impossible to confuse the offset a replace is removing with the one it is inserting. The old field aliased the docs map's key and was only safe because removal happened at the one chokepoint where a document dies; that constraint is gone. Done for secondary indexes too, not just `_id_`. That deletes the per-candidate `coll.docs.get(id)` in scan_sorted outright rather than replacing it with an `_id_` descent, and it is free on the write path because a replace already removes and reinserts every entry in every index. Consequences worth knowing: - lookup_eq/lookup_range/Plan.search yield u64. Those are values, immune to the tree mutation that invalidated the id slices they used to hand back -- which is why ttl_sweep_coll can drop the dupe-and-free dance it needed to survive `remove` freeing the key its entries pointed at. - One safety net is gone. A stale entry used to be swallowed by `docs.get(id) orelse continue`; now it resolves to superseded-but-parseable bytes the re-applied filter might accept. That trades an invisible under-approximation for a visible wrong answer, which is the better failure to have, but it is a trade. - A checkpoint may never renumber slab offsets (already recorded in PLAN §4): every index leaf now holds a physical one. `zig build fuzz` earned its keep immediately -- it caught the API break in all four B+tree harnesses, which `zig build test` cannot see. Benchmarks A/B'd at 256m on one harness, before and after: all rows flat. updateMany and deleteOne+insertOne first looked 10-13% slower, which three repeat runs showed to be single-sample noise (0.70/0.71/0.70 against 0.70). |
|||
| d867c37d32 |
commands: stop leaking the catalog lock on a nameless command
dispatch resolved the namespace *after* acquiring the catalog lock, and bailed
out with `orelse return` when either part was missing. A plain return is not an
error return, so it ran neither the errdefer nor the explicit unlocks after the
handler: the catalog lock was held, shared, for the life of the process.
`db.aggregate(...)` reaches it. That sends `{aggregate: 1}`, whose value is a
number, so str_arg returns null.
What made this hard to see is that a leaked *shared* lock is invisible to
readers. ping and listDatabases kept answering in microseconds, and the server
looked perfectly healthy from outside -- an external prober got `ok 15ms`
throughout. Only a write needing the catalog exclusive to create a collection
blocked, so the failure surfaced one command later, on a different connection,
as a client-side timeout with nothing to connect it to its cause. It cost three
invalid spec-test baselines before the driver's own command log showed an
insert sitting for exactly socketTimeoutMS against an idle engine.
Namespace resolution now happens before any lock is taken, and a missing name
is a BadValue reply instead of an empty document (which drivers render as the
uninformative "n/a").
Also fixes the aggregate path it exposed: a missing collection returned a reply
with no `ok` field, where MongoDB answers an empty cursor.
Tested by asserting both halves -- a real error reply, and that a following
write which creates a collection still completes. The second is the lock check.
Mutation-checked: reintroducing the leak reddens that test by name.
|
|||
| aee23cb028 |
index/db: enforce _id uniqueness through the _id_ index
_id uniqueness was a `coll.docs.contains` probe. The docs hashmap is going
away (PLAN A3), so it has to move to the _id_ tree -- and the tree answers
better, because it is keyed on bson.encode_key, which is canonical where
serialize_value is not. int32 1, int64 1 and double 1.0 are now one _id, as
they are in MongoDB (A4).
_id_ is built and checked first, so a write violating both it and a unique
secondary reports _id_, which is what MongoDB reports. It returns
error.DuplicateKey with `dup_index` left null, which is exactly what
commands.zig's E11000 rendering already treats as "the _id_ index", so the
wire-visible message is unchanged and that file needed no edit.
check_unique's exclude-self became optional and is null on an insert. That was
a latent bug of its own: a replace must ignore its own existing entries, but an
insert has none, and passing the document's id there hides a collision whose
entry carries that same id -- precisely the case _id_ exists to catch. Only
_id_ could reach it, since a secondary collision is between different
documents.
Two corrections found while doing this, both worth reading:
PLAN A4 claimed a database already holding {_id: int32 1} and {_id: int64 1}
loses one on reopen. It does not. Replay evicts through the docs map, keyed on
serialize_value, so both survive; the tree is bulk-built afterwards with
enforcement off, which tolerates duplicate keys and warns. The loss arrives
only with the commit that drops the map, and that is where it needs a
pre-flight scan. Amended.
dispatch_insert asserted only `ok: 1`, but a rejected document comes back as a
writeError alongside it -- so the mixed-type corpus silently shrank from ten
documents to nine when _id_ became unique, and every test over it still passed.
The helper now rejects writeErrors and asserts the inserted count; it caught
the shrink immediately. The corpus keeps an int64 _id on a distinct value, and
the collision it used to stand in for is asserted directly.
Also adds Index.lookup_exact, which the commands that currently probe the docs
map will need. Exact byte equality rather than cmp_prefix, because {a: 1}'s
encoding is a proper prefix of {a: 1, b: 2}'s and a prefix match would claim a
document is present when it is not.
Mutation-checked, all three red: unique=false on id_index; exclude=id_key on
insert; eql -> cmp_prefix in lookup_exact.
|
|||
| f61416f44a |
index/db: heap-allocate secondary indexes
Collection.indexes held Index by value, so orderedRemove memmoved the whole ~5 KB struct and every *Index already handed out referred to a different index afterwards -- a query plan's `index` field, or a slice into an index's promoted-key buffer. The collection's own bookkeeping stayed consistent, which is why nothing noticed: only a caller holding a pointer across a drop could see it, and no test did. The new test does, and it is mutation-checked against the by-value code that this commit replaces: holding pointers to b_1 and c_1, then dropping a_1, the b_1 pointer reads "c_1". Now orderedRemove moves 8-byte pointers, the surviving indexes do not move, and only the removed one is freed. M0 needs this independently: an Index will own a file mapping once the node arena moves into the data file, and copying one by value would duplicate that ownership. Not done, though the milestone plan listed it: moving Index's inline scratch and promo buffers out of the struct. Their stated purpose was to keep those 5 KB out of a file-resident Index and to stop the memmove -- but only the node arena and overflow slab become file-resident, not the Index metadata, and boxing already fixed the memmove. Moving them would be churn with nothing left to buy. |
|||
| 411a380d38 |
commands: fix a remote invalid free in aggregate $sort
Present since at least
|
|||
| d4c9b04f21 |
rename project to MultiforaDB
Prose and benchmark tables use MultiforaDB; the binary, the CLI usage line, the log-message prefix and the default database file use multiforadb. Two consequences worth noting: - build.zig.zon's fingerprint is derived from the package name, so it had to change with it (Zig refuses to build otherwise). A consumer pinning this package by fingerprint needs updating. - the default --db path is now multiforadb.log, and getCmdLineOpts reports it as dbpath. An existing mongo-lite.log has to be passed explicitly with --db. The e2e harness abbreviated the old name as ML_; that is now MFDB_, including the documented ML_BIN override (MFDB_BIN) and the scratch file names. MD_ (mongod) is untouched. compare-run.sh spawned the server by absolute path under a sandbox/mongo-lite directory that no longer exists; that block already runs from tests/e2e, so it uses a relative path now. The archived reports under tests/e2e/results/ keep the old name: they record what the old binary measured. |
|||
| 720540860a |
db/storage: close three data-loss paths in commit and compaction
Follow-up hardening on the group-commit work from |
|||
| c8d547fef5 |
db/commands: acknowledged writes reach the disk again
Three defects, each of which made the database lose data that had already
been acknowledged, or answer a client with a malformed reply.
- Engine.commit decided a writer was "already covered" by comparing
log.end_pos with the position of the last completed commit. Under block
framing an append leaves its bytes in the log's open in-memory block and
does not move end_pos -- only sealing does. So once the first commit had
set committed_end = end_pos, every later write command found itself
covered and returned without sealing or syncing anything. A no-op
deleteMany followed by insertMany(50) was acknowledged with the file
still 16 bytes (its header) and lost all 50 documents on kill -9, which
is precisely what e2e2's crash pair does. Coverage is now decided by
sequence number, which counts records rather than bytes on disk.
- Compaction read the new log's end position before syncing it, but the
sync is what seals the open block, and the seal is what moves end_pos
past it. Appends after a compaction therefore started inside the
compacted file's last block and overwrote it, so those documents were
gone at the next replay: e2e6's phase 2 ended with 1000 documents in
memory and 996 after a graceful restart.
- cmd_find returned early on a missing namespace without putting anything
in the reply, so a find on an unknown collection arrived at the driver as
a response with no `ok` field ("MongoServerError: n/a") instead of an
empty cursor. The other commands' missing-namespace paths were fine.
Verified with the unit suite in ReleaseFast/ReleaseSafe/Debug, the split
fuzzer, all six e2e suites (e2e6 back to 72/72) and the kill -9 crash pair
-- none of which passed beforehand -- plus 13 kill -9 runs over 1/2/8
connections with 1200 acknowledged inserts each and nothing lost.
tests/e2e/results/phase7.txt records the benchmark with the fixes in place:
no regression against phase6 (bulk 739 -> 753 MB/s, updateMany 1.9 -> 2.0
ms, RSS 547 -> 546 MB), and concurrent durable writes now measurable at
7.1k/15.0k/21.8k docs/s over 1/8/32 connections.
|
|||
| ecd28d9b26 |
engine: decompose the global lock; cross-connection group commit (roadmap item 5)
The single engine-wide reader/writer lock is replaced by a lock hierarchy, so writes to different collections no longer serialize on one mutex: - Collections are heap-allocated, so their addresses are stable while a command holds a collection lock (the maps only store pointers). - A catalog rwlock guards the database/collection maps: shared for every command (so a concurrent DDL cannot mutate the maps underneath it), exclusive for create/drop/dropDatabase. Each collection has its own rwlock; the ordering is always catalog -> collection -> log lock, never two collection locks at once (TTL sweep and compaction take collections one at a time). - Command dispatch acquires the catalog + target collection locks for the handler's duration, resolving the collection (creating it for writes) under the catalog lock; create/drop upgrade to the exclusive catalog lock. - Appends never fsync. Each write command's epilogue releases the collection lock, then commits once (seal + fsync) with a leader/follower group commit: the leader waits for writers mid-append (a pending counter) so its seal covers them, and followers whose records the seal covered skip their own fsync. Every acknowledged write is fsynced before its reply (crash pair verified); an unacknowledged write may vanish and a reader may observe a write before its fsync — ordinary w:1 j:true semantics instead of 'the log describes >= memory'. - Compaction snapshots collections without the log lock (so a concurrent writer holding one can always finish its append) and retries when a writer appended mid-snapshot (detected via the record seq), then swaps under the log lock — no deadlock. The compaction trigger moved to the command epilogue and the TTL monitor. - Engine.dup_index moved to the collection (per-command error paths). Also lands two B-tree edge-case fixes driven by tests that were in flight: a churned leaf full of dead bytes no longer splits with an empty right half (the leaf is repacked before splitting, and an emptied node's page is fully free again), and a slot-count split with all large records on one side shifts records between the halves until the new record fits. Plus a randomised fuzz test over key sizes (src/fuzz_split.zig) and the two regression tests. Measured (tests/e2e/results/phase6.txt): no regression on the single-connection benchmark; concurrent durable-insert throughput ~5.1k -> 12.5k docs/s from 1 -> 8 clients, ~14.8k at 32. Verified: unit suite in all three modes, all e2e suites, the kill -9 crash pair. |
|||
| 570900a6ef |
storage: byte documents in a per-collection slab (roadmap item 4)
Documents live as canonical BSON bytes in a segmented per-collection slab (fixed 8 MiB segments keep capacity slack under one segment); the docs map holds flat offsets that stay valid across segment growth, and removed documents leave garbage bytes until compaction rewrites. The per-document ArenaAllocator and its second full Pair-tree copy are gone. The matcher walks the stored bytes directly, skipping by length any field the filter does not name (a new bson byte-walker: element_key, skip_value, read_value with borrowed leaves, get_at, and a borrowed spine parse). The byte matcher is differential-tested against the tree matcher on a corpus and shares its operator logic. Stored documents are never materialized on the scan path or in aggregate $match; $group reads group keys and sums straight off the bytes. Sort, projection, findAndModify, updates and index entry generation use a borrowed spine into the slab (or the byte collector, which also replaced collect_values in build_entries). The compaction threshold now counts uncompressed data volume, since a compressed log would otherwise never trigger. Measured (tests/e2e/results/phase5.txt): server RSS 1979 -> 539 MB (2.4x smaller than MongoDB; phase1 baseline 2.0 GB), range-scan 22.5 -> ~12 ms (parity, best run faster than MongoDB), proj 4.1 -> 3.4 ms, createIndex parity. Verified: unit suite in all three modes with zero leaks, the crash pair, e2e6, and the stress/spill programs. |
|||
| 58914a69c3 |
index: ordered _id index (roadmap item 2)
Give every Collection an implicit _id_ index (a normal Index with keys
[_id: 1]) so _id equality, $in, ranges and sorts stop depending on the
docs-map hash or a full scan. Kept out of the secondary indexes list, so
listIndexes/dropIndexes/createIndex and the log format are unchanged (no
index_create record, no double listing) and e2e3.js passes unmodified.
Maintained in upsert through the same reserve-then-insert protocol as
the secondaries, removed in evict_doc, and rebuilt after replay by
build_all_indexes alongside them (never maintained mid-replay, so a
failed add can't leave the index under-approximating). index.plan now
takes it as a separate argument. Its keys are canonical
(bson.encode_key gives int32 1, int64 1 and double 1.0 identical bytes),
so the serialization-guarded docs-map fast path (plan_id,
value_fast_path_safe and friends) is deleted.
Measured (tests/e2e/results/phase3.txt): sort({_id:-1}).limit(20) 6.2 ->
2.4 ms (2.3x slower than MongoDB -> parity); integer/string _id point
lookups, $in and ranges verified against the tree. Unit suite in all
three optimize modes, the crash pair, e2e3/e2e4/e2e6.
|
|||
| 4aaa555563 |
index/commands: let an index supply the sort order
A sort ordered every match before discarding all but one page, even when
an index already held the candidates in exactly that order. The planner
now recognizes that case and the scan streams the page straight out.
An index provides the sort when the sort keys line up with the components
after the equality-pinned prefix (those are fixed to one value each, so
they do not affect the order of what follows) and every direction agrees
uniformly -- all the same way round, or all opposite, since the array can
only be read forwards or backwards. Multikey indexes are excluded: they
emit a document once per indexed value, so their order is not an order on
documents. So is an $in, whose disjoint ranges concatenate unordered.
Entries already come out of the array in key order, so forward scans were
sorted all along; what destroyed it was the dedupe pass sorting by id.
The conditions above are exactly the ones under which that pass is
skipped, so ordered output needs only reversing for a backward scan.
evaluate_index gains a plan shape it did not have: a full index scan when
the ordering is the reason to use it. Without that, find({}).sort(...)
was unreachable -- an empty filter yields no clauses and the planner bailed
before looking at any index. It is guarded to only appear when the sort is
satisfied, since otherwise scanning the docs map directly is cheaper.
The early stop had to move into the scan, which is the only place that
knows whether the order came from an index: a limit is a valid page
boundary without a sort, or with one an index supplies, and otherwise
means nothing. Getting this wrong the other way -- limiting first and
re-scanning -- would have doubled the work for every unindexed sort.
find({}).sort({k: 1}).limit(20) over 65,536 x 16 KB documents:
4.0ms -> 1.0ms
sort({_id: -1}) is unchanged at 4.3ms: nothing ordered covers _id yet.
Checked against the definition rather than by example: eleven query shapes
-- forward, backward, skip, unlimited, filtered, equality-prefixed both
directions, compound, directions the index cannot serve, and a range --
each compared through the real driver against the page of the equivalent
full sort.
Verified: 78 unit tests under ReleaseFast and ReleaseSafe, e2e
29/16/17/3/2, the crash pair, e2e6 72/72.
|
|||
| 75e412a4af |
query/commands/wire: trim the scan and request paths
Matching allocated an ArrayList per filter field per candidate document,
on the process-wide allocator, to hold what is almost always a single
value. Candidates now collect into a stack buffer that spills to the heap
only for arrays: measured 15.7 -> 12.0ms on a 65,536-document range scan.
The OOM-propagation test moves with it. Its point is that a failed
collection must surface as an error rather than an empty candidate list,
which would make $ne and $exists:false report a match -- a wrong answer
rather than a failed one. That invariant still holds on the spill path, so
the test now uses an array long enough to reach the allocator, and a new
test pins the flip side: the common single-value match now completes
correctly even when the allocator always fails, because it never calls it.
Query operators were dispatched by a chain of up to fourteen mem.eql per
value per document, with $gt/$gte/$lt/$lte re-comparing the operator name
inside the loop over candidate values. Names resolve to an enum once per
filter field. Command dispatch likewise walked a 30-entry table comparing
strings; it is a comptime StaticStringMap now.
Each request built a fresh reply arena and handed its pages straight back.
One reply per connection, reset between requests, keeps them.
countDocuments() arrives as [{$match: F}?, {$group: {_id: <literal>,
n: {$sum: 1}}}], which the general path answered by materializing every
matching document and discarding them all. It is now recognized and
answered from a counting scan: countDocuments({}) 2.3 -> 1.5ms.
The detector is deliberately conservative -- grouping by "$field", summing
a field, an unmodelled accumulator or any extra stage all fall through to
the general path, since those need the documents themselves. A unit test
pins each accept and reject, and the whole count path was checked against
the general one through the real driver, including the shapes that must
not take it.
The filtered range-scan row does not move: it is bound by walking 65,536
documents that each live in their own arena, not by the matcher. That is
Phase 4 work.
Verified: 77 unit tests under ReleaseFast and ReleaseSafe, e2e
29/16/17/3/2, the crash pair, e2e6 72/72.
|
|||
| 9eecb5092c |
query/commands: top-k sort selection and an allocation-free decorate pass
sort+limit ordered the entire result set to return one page: 65,536
documents sorted to hand back 20. Two independent costs.
The decorate pass built, per document per sort key, an ArrayList of every
value at the path -- but the comparator only ever reads element 0. Added
first_value_at, which mirrors collect_values' traversal exactly (same
order, same depth cutoff) and stops at the first hit, and moved the
decorated values into one flat allocation. That equivalence is the whole
correctness argument, so it is pinned by a test covering dotted paths,
arrays of documents, numeric element addressing, repeated keys, missing
paths and the depth cutoff.
sort_docs_top_k keeps a k-element max-heap instead of ordering
everything: one comparison against the heap root per document, and only
the survivors are ever sorted. cmd_find uses it when the page is at most
a quarter of the matches, where the heap's bookkeeping still pays for
itself, and falls back to a full sort otherwise. It leaves docs[k..]
unordered, which is safe because the page is a prefix of the first k.
cmd_aggregate's $sort is deliberately untouched: a later stage can read
the whole stream, and top-k would silently corrupt the tail.
find({}).sort({_id:-1}).limit(20) over 65,536 x 16 KB documents:
baseline 40.0ms
decorate only (top-k disabled) 23.9ms
decorate + top-k 4.3ms
Correctness checked end to end as well: the limited page is identical to
the prefix of the equivalent full sort. The top-k test compares against a
full sort across ascending, descending and compound keys, for k of 1, 2,
20, n-1, n and n+1, over data with heavy ties; verified it fails when the
heap's child comparison is inverted.
|
