M3: the positional operators and arrayFilters #7

dev · 2026-08-10T17:49:25Z

dev commented

2026-08-10 17:49:25 +00:00

The positional operators, against the gate recorded for them last PR.

$[], $[<identifier>] and $ stop being refused and start being walked, and
arrayFilters reaches the update that names it.

The shape

A path with a positional segment names a set of concrete paths rather than
one: y.$[].b on a two-element array names y.0.b and y.1.b. resolve
expands it against the document at hand and every operator then walks paths it
already knew how to walk. Nothing below resolve knows a positional segment
exists, which is why $set, $inc, $unset, $push and $pull all get it at
once rather than one at a time.

The static half -- which identifier binds to which array filter, whether a
segment may sit in first position, whether a filter went unused -- depends only
on the update and the filters, so it runs once in validate, before the scan.
That placement is load-bearing in both directions and a test pins each:

an array filter the update never uses is refused (9) even when the query
matches nothing at all;
an identifier nothing binds is refused (2) before a single document is read.

Measured, not recalled

56 shapes run against mongod 8.3.7. Three changed what the code does:

$[] in first position answers the array filter identifier message, not
the $ one -- mongod treats the two spellings as one check.
an array filter may have several top-level fields as long as they all name
the same identifier: {i.b: 3, i.c: 1} is legal, {i.b: 3, j.b: 1} is not.
So the check is on the name, not on the count of fields.
$rename refuses a positional path on either end, with its own message for
each, so it keeps the plain split rather than resolving.

Also settled: an identifier is [a-z][a-zA-Z0-9]* (aB2 yes, Ab/a_b/1x
no); $unset of an element leaves a null rather than shortening the array; a
scalar element with more path below it is PathNotViable (28), which without the
check would have had the walk replace the 7 at y.1 with {b: 9} -- a
smaller copy of the destruction this work replaced.

Numbers

	before	after
`tests/spec/positional/`	15 pass / 36 fail	51 / 0
pinned crud scorecard	204 pass / 87 fail	218 / 73
unit tests	208	222, ReleaseFast and ReleaseSafe

The scorecard delta is the fourteen arrayFilters cases across five files and
nothing else -- which is what a corpus recorded from mongod is for.
zig build fuzz 83/83; aggregation corpus 70/0; the full e2e matrix and
crash-fuzz green.

Divergences, recorded rather than papered over

Three answers still differ from mongod, all in PLAN §6:

$ with two predicates on one array. {"y.b": 3, "y.c": 2} matches
[{b: 3, c: 1}, {b: 1, c: 2}] without either element satisfying both, and the
document still matched. mongod writes element 1, this writes element 0.
mongod's answer is an artefact of which predicate last wrote its match
position -- reversing the two predicates still gives 1 -- so there is no rule
here to copy, only a behaviour to record.
An array filter with a top-level $and/$or. mongod accepts it and finds
the identifier inside; this refuses with 9.
A literal index into a scalar element. $set: {"y.0.b": 1} on y: [null]
is PathNotViable on mongod; here it creates. The positional walk refuses it,
the plain indexed path does not. One rule reached two ways, and only one way
agrees -- worth its own commit, and it needs its own measurements first,
because null-padding past the end of an array is a legal creation.

Mutations

Five run. Four redden as their comments promise. The fifth did not: the comment
claimed moving update.validate below scan_matching would break the test, and
it does not -- the call still sits above the zero-match branch. What breaks it
is deleting the standalone call. Fixed in 1482df8; a mutation note nobody ran
is worth less than no note.

The positional operators, against the gate recorded for them last PR. `$[]`, `$[<identifier>]` and `$` stop being refused and start being walked, and `arrayFilters` reaches the update that names it. ## The shape A path with a positional segment names a *set* of concrete paths rather than one: `y.$[].b` on a two-element array names `y.0.b` and `y.1.b`. `resolve` expands it against the document at hand and every operator then walks paths it already knew how to walk. Nothing below `resolve` knows a positional segment exists, which is why `$set`, `$inc`, `$unset`, `$push` and `$pull` all get it at once rather than one at a time. The static half -- which identifier binds to which array filter, whether a segment may sit in first position, whether a filter went unused -- depends only on the update and the filters, so it runs once in `validate`, before the scan. That placement is load-bearing in both directions and a test pins each: - an array filter the update never uses is refused (9) **even when the query matches nothing at all**; - an identifier nothing binds is refused (2) before a single document is read. ## Measured, not recalled 56 shapes run against mongod 8.3.7. Three changed what the code does: - `$[]` in first position answers the *array filter identifier* message, not the `$` one -- mongod treats the two spellings as one check. - an array filter may have several top-level fields as long as they all name the same identifier: `{i.b: 3, i.c: 1}` is legal, `{i.b: 3, j.b: 1}` is not. So the check is on the name, not on the count of fields. - `$rename` refuses a positional path on *either* end, with its own message for each, so it keeps the plain split rather than resolving. Also settled: an identifier is `[a-z][a-zA-Z0-9]*` (`aB2` yes, `Ab`/`a_b`/`1x` no); `$unset` of an element leaves a null rather than shortening the array; a scalar element with more path below it is PathNotViable (28), which without the check would have had the walk replace the `7` at `y.1` with `{b: 9}` -- a smaller copy of the destruction this work replaced. ## Numbers | | before | after | |---|---|---| | `tests/spec/positional/` | 15 pass / 36 fail | **51 / 0** | | pinned crud scorecard | 204 pass / 87 fail | **218 / 73** | | unit tests | 208 | 222, ReleaseFast and ReleaseSafe | The scorecard delta is the fourteen `arrayFilters` cases across five files and nothing else -- which is what a corpus recorded from mongod is for. `zig build fuzz` 83/83; aggregation corpus 70/0; the full e2e matrix and `crash-fuzz` green. ## Divergences, recorded rather than papered over Three answers still differ from mongod, all in PLAN §6: - **`$` with two predicates on one array.** `{"y.b": 3, "y.c": 2}` matches `[{b: 3, c: 1}, {b: 1, c: 2}]` without either element satisfying both, and the document still matched. mongod writes element 1, this writes element 0. mongod's answer is an artefact of which predicate last wrote its match position -- reversing the two predicates still gives 1 -- so there is no rule here to copy, only a behaviour to record. - **An array filter with a top-level `$and`/`$or`.** mongod accepts it and finds the identifier inside; this refuses with 9. - **A literal index into a scalar element.** `$set: {"y.0.b": 1}` on `y: [null]` is PathNotViable on mongod; here it creates. The positional walk refuses it, the plain indexed path does not. One rule reached two ways, and only one way agrees -- worth its own commit, and it needs its own measurements first, because null-padding past the end of an array *is* a legal creation. ## Mutations Five run. Four redden as their comments promise. The fifth did not: the comment claimed moving `update.validate` below `scan_matching` would break the test, and it does not -- the call still sits above the zero-match branch. What breaks it is deleting the standalone call. Fixed in `1482df8`; a mutation note nobody ran is worth less than no note.

dev added 4 commits 2026-08-10 17:49:25 +00:00

update: a positional path resolves to the elements it names a05874d597

`$[]`, `$[<identifier>]` and `$` stop being refused and start being walked.
A path with a positional segment names a *set* of concrete paths rather than
one -- `y.$[].b` on a two-element array names `y.0.b` and `y.1.b` -- so
`resolve` expands it against the document at hand and every operator then
walks paths it already knew how to walk. Nothing below `resolve` knows a
positional segment exists, which is why `$set`, `$inc`, `$unset`, `$push` and
`$pull` all get it at once.

The static half -- which identifier binds to which array filter, whether a
segment may sit in first position, whether a filter went unused -- depends
only on the update and the filters, so it runs once in `validate` before any
document is touched. The document-dependent half is resolution itself: an
absent or non-array path is a refusal there, because array updates address
what is there rather than creating it, unlike `$set` on a plain path.

Codes and messages measured on mongod 8.3.7, not recalled. Sixteen shapes
were run; the ones that changed what this commit does:

  - `$[]` in first position answers the *array filter identifier* message,
    not the `$` one -- mongod treats the two spellings as one check.
  - an array filter may have several top-level fields as long as they all
    name the same identifier: `{i.b: 3, i.c: 1}` is legal, `{i.b: 3, j.b: 1}`
    is not. So the check is on the name, not on the count.
  - an identifier is `[a-z][a-zA-Z0-9]*`: `aB2` yes, `Ab`, `a_b`, `1x` no.
  - `$rename` refuses a positional path on *either* end, with its own message
    for each, so it keeps the plain split rather than resolving.
  - `$unset` of an element leaves a null in its place rather than shortening
    the array -- the same answer `$unset: {"y.0": ""}` already gave.
  - a scalar element with more path below it is PathNotViable (28), not a
    field to create. Without that check the walk would hand `y.1.b` to
    `set_path` and it would replace the `7` at `y.1` with `{b: 9}` -- a
    smaller copy of the destruction this whole walk replaced.

One divergence, measured and deliberate: `{"y.b": 3, "y.c": 2}` matches
`[{b: 3, c: 1}, {b: 1, c: 2}]` without either element satisfying both, and
`$` then picks element 1 on mongod and element 0 here. mongod's answer is an
artefact of which predicate last wrote its match position; guessing at it
would be worse than recording it. PLAN §6.

The command handlers still pass neither the query nor any array filters, so
over the wire this is `$[]` working and the other two refusing -- for the
right reason and with the right code, which they did not before. The plumbing
is the next commit.

221/221 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz.
Positional corpus 15 -> 23 of 51.

commands: arrayFilters and the query reach the update 200228b0cb

The engine landed last commit with nothing feeding it: `$[<identifier>]` had
no filters to bind and `$` had no query to resolve against, so both refused
correctly and uselessly. This is the plumbing.

`arrayFilters` is read per update statement on `update` and once on
`findAndModify`, which is where each command carries it. Only the *shape* is
checked here -- an array, of documents, TypeMismatch (14) with mongod's own
field names for either -- because which identifier a filter names, whether it
is spelled legally and whether the update ever uses it all need the update's
paths, and those belong to `update.validate`.

`validate` is called before `scan_matching`, not inside the per-document
loop, and that placement is load-bearing in both directions: an array filter
the update never uses is refused (9) even when the query matches nothing at
all, and an identifier nothing binds is refused (2) before a single document
is read. Both measured; a test pins each, with the mutation that reddens it
named in the comment.

Positional corpus 23 -> 51 of 51, the gate green.
Pinned crud scorecard 204 -> 218 pass, 87 -> 73 fail: the whole arrayFilters
cluster, which until two commits ago was answering ok: 1 having replaced the
array with a document keyed by the path segment's literal text.

222/222 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz.

plan/spec: the positional gate goes green 1892cb7969

Scorecard 204 -> 218 pass, 87 -> 73 fail: the fourteen `arrayFilters` cases
across five files, which is the whole of what the two implementation commits
were expected to move and nothing else. Positional corpus 51/51.

The three divergences measured on the way are written into PLAN §6 rather
than left in commit messages: `$` with two predicates on one array that no
element satisfies together, an array filter with a top-level `$and`/`$or`,
and a literal index into a scalar element -- the last being the one place the
positional walk and the plain indexed path now answer differently, which is
worth a commit of its own and needs its own measurements first.

The design review gets an outcome note, since two of its guesses were wrong
and the corpus is where that was settled.

commands/tests: name the mutation that actually reddens the arrayFilters test 1482df891b

The comment claimed moving `update.validate` below `scan_matching` would
redden it. Ran the mutation: it does not -- the call still sits above the
zero-match branch, so it still runs. What reddens it is deleting the
standalone call and leaving the check to `apply`, which runs once per matched
document and so never at all when nothing matched. A mutation note that has
not been run is worth less than no note.

dev merged commit 1482df891b into main

2026-08-10 17:49:33 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dev/MultiforaDB#7