M3: the update operators, and 's modifiers #8
Reference in New Issue
Block a user
Delete Branch "m3-operators"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The eight update operators PLAN §3 names for M3, plus
$push's modifiers.Corpus first, then the code under it -- the same order as M2.5 and the
positional work.
Why a corpus rather than a list of TODOs
A probe running the identical update against mongod 8.3.7 and this server
found all eight operators answering
bad update, code 2, one message forevery question. It also found the part that is not a missing feature:
$push's$slice,$positionand$sortwere parsed, accepted anddropped.
{$each: [3, 4], $slice: -3}appended both values, slicednothing, and answered
ok: 1withmodifiedCount: 1.A missing operator is an error the client can see. A modifier that is accepted
and ignored is the same class of wrong answer the positional operators were --
the client asked for one thing and was told it got it.
Numbers
tests/spec/operators/(102 cases)tests/spec/positional/tests/spec/aggregate/zig build fuzz83/83; the full e2e matrix andcrash-fuzzgreen. The pinnedcorpus not moving is the expected result, not a disappointment: it barely
mentions these operators, which is why the new corpus exists.
What measuring settled
Every one of these was recorded from mongod, and several are the opposite of
the obvious reading:
{$mul: {gone: 5}}writes 0, not 5.$min/$maxare not numeric operators. They compare in BSON canonicalorder, so
{$min: {s: 5}}ons: "b"writes 5, and a missing field isalways written.
$addToSetcompares documents whole, field order included --{a:1,b:2}and{b:2,a:1}are two values -- but2and2.0are one.$eachthere are no$pushmodifiers at all:{$slice: 1}isa document to push. With it, the order is position, then sort the whole
array, then slice.
$slice: -nkeeps the last n.$currentDate: {d: false}still writes a date; the boolean says "a date",not "whether".
$setOnInsertmay write_idon an insert, where$setmay not: adocument nobody has yet has no identity to change.
$popagainst a non-array is TypeMismatch (14) where the identical mistakeunder
$addToSetis BadValue (2), and$eachthat is not an array is 14under
$addToSetand 2 under$push. Three codes for one shape, nonederivable from the others.
Two refusals this server did not have
is a prefix of the other at a segment boundary.
a.banda.bbshare astring prefix and are not a conflict, which is the mutation the test
names.
BadValue. What matters is the optionnobody took: skipping an unknown operator makes a typo an update the client
believes ran.
A bug that was nobody's operator
The corpus found that an upsert never reported the
_idit generated.Engine.insertwrites a generated_idinto the bytes and leaves thecaller's tree without it, so
updateOne(..., {upsert: true}).upsertedIdcameback null and
findOneAndUpdatewithreturnDocument: afterreturned adocument with no
_id-- the only name a client has for a document it hasnever seen. Fixed in
build_upsert_doc, after the operators run because$setOnInsertmay supply the_iditself. This is the first corpus here thatupserts into an empty collection and then looks at what came back.
One existing test was wrong
It packed
$set: {new: 5}and$rename: {new: "renamed"}into one update andpassed only because there was no conflict check. mongod refuses that with 40 --
measured, then the test split in two.
Left open, in PLAN §6
$bitis not implemented (mongod has it, no driver intests/sends it, andit is not in the M3 row);
$sortinside$pushaccepts only a one-field keydocument where mongod allows several; the conflict check stops after 64
distinct paths in one update rather than refusing a legal wide one.
PLAN §3 lists eight operators for M3 -- `$setOnInsert`, `$addToSet`, `$mul`, `$min`, `$max`, `$pop`, `$pullAll`, `$currentDate` -- and the pinned crud corpus says almost nothing about any of them. A probe running the identical update against mongod 8.3.7 and this server found all eight answering `bad update`, code 2, one message for every question. It also found the thing this directory exists for: `$push`'s `$slice`, `$position` and `$sort` are **silently ignored**. `{$each: [3, 4], $slice: -3}` appends both values, slices nothing, and answers ok: 1 with modifiedCount: 1. A missing operator is an error the client can see; a modifier that is parsed, accepted and then dropped is the same class of wrong answer the positional operators were. 103 cases in six files, 18 pass / 84 fail -- red by construction, like `tests/spec/aggregate/expressions.json` at 1/26 and the positional corpus at 15/36. Inputs authored in `sources/`, every expectation measured. Two things here cannot be recorded as values, and both become a `$$type` assertion rather than being left out: a `$currentDate` field is whatever the clock said (named per case in `volatile`, so a real stored date can still be pinned one day), and an upsert that inserts gets a generated ObjectId (that one automatic -- no source authors an ObjectId). Everything else is compared exactly. Files are canonical extended JSON, which the runner already parses that way: `$mul` overflowing an int32 produces an int64, and writing `4000000000` as a bare number would not have said so. What recording it settled, none of it guessable: - `$mul` of a missing field writes **0**, not the operand; of a non-numeric field, or by one, TypeMismatch (14). - `$min`/`$max` are not numeric operators. They compare in BSON canonical order, so `$min: {s: 5}` on `s: "b"` writes 5, and a missing field is always written. - two operators writing one field is **ConflictingUpdateOperators (40)** -- `$min`+`$max`, `$set`+`$inc`, `$setOnInsert`+`$set`. A whole error class this server does not have. - `$addToSet` compares documents whole, **field order included**: `{a:1,b:2}` and `{b:2,a:1}` are two values. But `2` and `2.0` are one. - `$push` modifiers are only modifiers when `$each` is there: `{$slice: 1}` alone is a value to push. With it, the order is position, then sort the whole array, then slice. - `$currentDate` with `false` still writes a date. - `$setOnInsert` may write `_id` on an insert, where `$set` may not. - an unknown modifier is FailedToParse (9), not BadValue. One case was authored and then removed: `{b: 1, $set: {c: 1}}` never reaches a server -- the driver rejects it -- so there was no answer to record and the case would have asserted nothing.Three operators, two shapes. `$mul` joins `$inc` in `op_arith` because they differ only in the operation; `$min` and `$max` are not numeric operators at all and get their own. Measured, and each row is a rule that would have been guessed wrong: - `{$mul: {gone: 5}}` writes **0**, not 5. An absent field starts from zero under both operators, which is the identity for one and the annihilator for the other, and mongod picks zero for both. - `$min`/`$max` compare in BSON canonical order, so `{$min: {s: 5}}` on `s: "b"` writes 5 -- a number ranks below a string -- and `{$max: {a: 1}}` on `a: null` writes 1. An absent field is always written: there is nothing to be smaller or larger than. - an int32 product that does not fit widens to int64, the same ladder `numeric_add` already climbed. `$inc` changes answer with them: a non-numeric field or operand was `InvalidUpdate` -> BadValue (2), and mongod answers TypeMismatch (14) with a different sentence for each side. So two errors rather than one, and `$inc` gets the codes it should always have had. numeric.json 0/21 -> 20/21. The one left is `$min` and `$max` on the same field, which is ConflictingUpdateOperators (40) -- a whole error class this server does not have yet, and its own commit. 228/228 unit tests.`$addToSet`'s identity is `bson.compare` equality, which is already exactly mongod's: an int32 `2` and a double `2.0` are one value, and `{a: 1, b: 2}` and `{b: 2, a: 1}` are two, because `compare_docs` walks the pairs positionally and tie-breaks on the key. Candidates are checked against the array as it grows, so a `$each` holding the same value twice adds it once. `$pullAll` is `$pull`'s neighbour and its opposite: `$pull` takes a predicate, `$pullAll` takes values compared whole. `{$pull: {t: {a: 1}}}` removes elements *having* `a: 1`; `{$pullAll: {t: [{a: 1}]}}` removes elements that *are* `{a: 1}`. Sharing the comparison would have been the natural mistake and is what the test mutates to check. `$pop` is the small one, and its refusals are the measured part: an empty array and an absent field are no-ops, an argument that is not 1 or -1 is FailedToParse (9), and a non-array field is TypeMismatch (14) -- where the identical mistake under `$addToSet` and `$pullAll` is BadValue (2). Three codes for one shape, none of them derivable from the others. `$each` that is not an array is 14 under `$addToSet` and 2 under `$push`, measured on both. array-ops.json 2/26 -> 26/26. 234/234 unit tests.`$slice`, `$position` and `$sort` were parsed, accepted and dropped. `{$each: [3, 4], $slice: -3}` appended both values, sliced nothing and answered ok: 1 with modifiedCount: 1 -- the same class of wrong answer the positional operators were, and the reason `tests/spec/operators/` exists rather than a list of TODOs. Measured, and the order is the whole of it: insert at `$position`, then `$sort` the array *including* the new elements, then `$slice` the result. - `$slice: n` keeps the first n, `$slice: -n` the **last** n. That half is what a capped log depends on and is the one easy to write backwards; the test mutates exactly it. - `$position` counts back from the end when negative, and clamps at the front rather than wrapping. - `$sort: 1` orders whole elements in BSON order; `$sort: {a: 1}` orders on a field of them, and an element without it sorts as null -- the rank a missing field has everywhere else here. - **without `$each` there are no modifiers at all**: `{$push: {t: {$slice: 1}}}` pushes the document `{$slice: 1}` as a value. That is what makes `$each` the flag rather than a member of the set, and it is measured, not reasoned. An unknown `$`-prefixed key beside `$each`, or a `$slice`/`$position` that is not a number, is BadValue -- refused rather than ignored, which is the point. push-modifiers.json 6/21 -> 21/21. 240/240 unit tests.`$setOnInsert` is `$set` on the branch that inserts and nothing at all on the branch that updates, so `Options` grows the one bit that says which -- and `build_upsert_doc` is the only caller that sets it. It does not go through `op_set` because of the measured exception: writing `_id` is allowed on a document being built and refused on one being rewritten. A document nobody has yet has no identity to change. `$currentDate` reads a clock, and the clock is a parameter. There is no fallback to a global one: this file has no `io` to reach the real clock through, and the handlers pass the same `std.Io.Timestamp` that `ttl_sweep` and the cursor sweep already read. A caller that forgets gets the epoch -- deterministic and obviously wrong -- rather than something that changes between runs. Measured: `$currentDate: {d: false}` writes a date. The boolean says "a date", not "whether", and either value means the same thing. `{$type: "timestamp"}` writes a BSON timestamp, seconds in the high 32 bits and an ordinal in the low ones; mongod fills the ordinal from the oplog, a standalone has none, so it is 1. current-date.json 3/11 -> 11/11, set-on-insert.json 0/9 -> 7/9. The two left are not `$setOnInsert`'s: one is ConflictingUpdateOperators, and the other is a pre-existing bug the corpus found -- an upsert never reports the `_id` it generated, because `Engine.insert` writes it into the bytes and not back into the caller's tree. `updateOne(..., {upsert: true}).upsertedId` is null here and an ObjectId on mongod. Next commit. 244/244 unit tests.Two refusals this server did not have, both measured. **ConflictingUpdateOperators (40).** Two paths in one update where either is a prefix of the other at a segment boundary leave the result depending on which operator ran first, so mongod refuses rather than picking an order: `{$set: {a: 2}, $inc: {a: 1}}`, `{$min: {a: 2}, $max: {a: 9}}`, `{$set: {a: 2, "a.b": 3}}`, and `$rename` counts both its ends. Siblings are fine, and the segment boundary is what makes them fine -- `a.b` and `a.bb` share a string prefix and decide nothing about each other, which is the mutation the test names. The check is quadratic in the number of paths and bounded at 64, so it stays on the stack. Past the bound it stops checking rather than refusing: missing a conflict in a 65-field update is a better answer than refusing a legal one, and no driver writes updates that wide. **Unknown modifier (9).** A name not in the operator table was `InvalidUpdate` -> BadValue; mongod answers FailedToParse and names it. What matters is the option nobody took: skipping an unknown operator would make a typo an update the client believes ran. A known operator handed something that is not a document of fields is the same code. One existing test changed answer and was wrong: it packed `$set: {new: 5}` and `$rename: {new: "renamed"}` into one update and passed only because there was no conflict check. mongod refuses that with 40 -- measured, then the test split into two updates. operators corpus 95/102 -> **102/102**. The pinned crud corpus, the positional corpus and the aggregation corpus are unmoved at 218/73, 51/0 and 70/0. 247/247 unit tests.