M3: pipeline-style updates #9
Reference in New Issue
Block a user
Delete Branch "m3-pipeline-updates"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
An update document may be an array of aggregation stages instead of a
document of operators. This is the one M3 line whose gate was already in the
pinned crud corpus, and it moves that corpus more than anything else in the
milestone has.
Numbers
tests/spec/operators/(7 files)tests/spec/positional/tests/spec/aggregate/zig build fuzz83/83; the full e2e matrix andcrash-fuzzgreen.The +10 is the five
*-pipelinefiles, the four-rawdatafiles that alsowrite pipelines, and
findOneAndUpdate-comment, which is two more pipelinecases with a comment beside them.
The corpus went into
tests/spec/operators/Not an eighth directory: a pipeline is another shape of update document, and
it shares the recorder, the
$$typemasking and the README. 23 cases,recorded red at 0/23.
Three shapes were authored and removed —
{b: 1, $set: {...}}, an emptypipeline, and a pipeline holding a non-document. The driver rejects all three
before they reach a server, so there is no server answer to record and the
cases would have asserted nothing.
What decided the design
The
_idsurvives every stage.$replaceRoot: {newRoot: "$t"}drops itand
$project: {_id: 0}asks to, and it comes back either way, because apipeline update rewrites a document rather than replacing one document with
another. A stage that sets it to a different value is ImmutableField (66);
restating the same one is fine. The mutation that reddens the test is deleting
the restore -- the document becomes unfindable by the id it is stored under.
Six stages are allowed:
$addFields/$set,$project,$unset,$replaceRoot/$replaceWith. Each rewrites one document into one document,which is the property that makes them usable here --
$matchcould drop it,$groupand$unwindcould change how many there are,$sortmeans nothingto one. Those four are refused by name with InvalidOptions (72), and a
name that is no stage at all with 40324. "Not here" and "not at all" are
different things to be told, and mongod tells them apart.
Two things that fell out
$replaceWithdid not exist in this server at all. It is$replaceRootwith the expression in place of the
{newRoot: ...}wrapper, and since thestage compiler is shared,
aggregategets it too.EvalCtx.collbecomes optional. Only an.offsetsstream reads the slab; a.docsone never touched the field. That is what lets the upsert path run apipeline over a document held in memory, before the collection it will be
inserted into exists.
And one refusal
arrayFiltersbeside a pipeline is FailedToParse (9), not ignored: anidentifier a pipeline cannot spell would make the update silently different
from the one the client wrote.
An update document may be an *array* of aggregation stages instead of a document of operators. The pinned crud corpus has five cases, one per command, and not one of them is a refusal -- so which stages are allowed, what happens to `_id`, and what an upsert does are all unmeasured there. 23 cases into the existing operator corpus rather than an eighth directory: this is another shape of update document, and it shares the recorder, the masking and the README. Recorded red, 0/23. What it settled: - **`_id` always survives**, through `$replaceRoot: {newRoot: "$t"}` and through `$project: {_id: 0}` alike. A stage that sets it to a *different* value is ImmutableField (66); restating the same one is fine. - `$match`, `$group`, `$sort` and `$unwind` are real stages refused specifically here: InvalidOptions (72), "$X is not allowed to be used within an update". A name that is no stage at all is 40324, and two stages packed into one array element is 40323. - `arrayFilters` beside a pipeline is FailedToParse (9). - an upsert runs the pipeline over the document the filter implies. Three shapes were authored and removed: `{b: 1, $set: {...}}`, an empty pipeline, and a pipeline holding a non-document. The driver rejects all three before they reach a server, so there is nothing to record.`u` (and `findAndModify`'s `update`) may be an array of aggregation stages instead of a document of operators. The stages themselves already existed -- M2 built `compile_rewrite` and `apply_rewrite` for `aggregate` -- so this is mostly about which of them are allowed here and what happens around them. Six are: `$addFields`/`$set`, `$project`, `$unset`, `$replaceRoot`/ `$replaceWith`. Each rewrites one document into one document, which is the property that makes them usable: `$match` could drop it, `$group` and `$unwind` could change how many there are, and `$sort` means nothing to one. mongod refuses those four by name with InvalidOptions (72) and a name that is no stage at all with 40324, and the difference is worth keeping -- "not here" and "not at all" are different things to be told. `$replaceWith` did not exist here at all and now does, in `compile_rewrite`, so `aggregate` gets it too: it is `$replaceRoot` with the expression in place of the `{newRoot: ...}` wrapper, one stage under two spellings. **The `_id` survives every stage.** `$replaceRoot: {newRoot: "$t"}` drops it and `$project: {_id: 0}` asks to, and it comes back either way, because a pipeline update rewrites a document rather than replacing one document with another. A stage that sets it to a *different* value is ImmutableField (66); restating the same one is fine. Measured on all six stages, and the mutation that reddens the test is deleting the restore -- the document becomes unfindable by the id it is stored under. `EvalCtx.coll` becomes optional, which is what lets the upsert path run a pipeline over a document held in memory before the collection it will be inserted into exists. Only an `.offsets` stream reads the slab; a `.docs` one never touched the field. `arrayFilters` beside a pipeline is FailedToParse (9), not ignored: an identifier a pipeline cannot spell would make the update silently different from the one the client wrote. pipeline.json 0/23 -> 23/23, operator corpus 125/125. Pinned crud scorecard **218 -> 228 pass, 73 -> 63 fail**: the five pipeline files, the four `-rawdata` files that use pipelines, and `findOneAndUpdate-comment`, which is two more pipeline cases. 249/249 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz.