tests/spec: a recorded corpus for the positional operators
M3's gate as named -- "remaining crud coverage; e2e3/e2e4 green" -- cannot
see this work. e2e3 and e2e4 contain zero positional paths, and the pinned
crud corpus covers `$[<identifier>]` only: no `$[]` case, no bare `$` case
anywhere in it. Both stayed green through a bug that replaced an array with
`{"$[i]": {...}}` and answered ok: 1. So M3 brings its own corpus, built the
way M2.5's was: inputs authored in `sources/`, every expectation recorded
from mongod 8.3.7, run through the shared runner with `--suite-dir`.
51 cases across the three spellings. It stands at 15 pass / 36 fail against
the refusal, which is the intended shape -- `expressions.json` was recorded
at 1 pass / 26 fail before the evaluator and is green now. The 15 that pass
are refusals where this server's code already matches; of the 36, 31 are the
constructs answering "not implemented" and 5 are refusals whose code
differs, four of them arrayFilters validation this server cannot do because
it never parses the option.
Every case records its `outcome`, refusals included. That is deliberate and
it is the whole point of the file: a refusal that left the document mangled
is indistinguishable from a clean one in `expectError` alone, and a mangled
document is what this corpus exists to catch.
What recording it settled, none of it guessable, the first contradicting
what the design review assumed:
- `y.$[i].c.$[i].d`, one identifier reused at two levels, is **accepted**
-- not a duplicate-identifier error
- `$[]` over an empty array is a no-op with modifiedCount 0
- `$[]` over an array with a non-document element is error 28, where every
other path failure here is 2
- a positional segment never creates: a missing or non-array path is an
error, where `$set: {'a.b': 1}` would construct one
- an upsert gets no special case -- it fails for the same reason
- `$` writes only the first matching element, and is refused when the query
never touched the array
- any of the three in first position is refused, as is `$` twice in a path
- `arrayFilters` alongside a replacement is ignored rather than refused
A case needing its own documents reseeds through operations rather than
`initialData`, because the format's `initialData` is per file and the runner
seeds it once per test. That keeps one file per operator instead of one file
per document shape.
crud scorecard unchanged at 204/87, aggregation corpus still 70/0.
This commit was merged in pull request #6.
This commit is contained in:
119
tests/spec/positional/README.md
Normal file
119
tests/spec/positional/README.md
Normal file
@@ -0,0 +1,119 @@
|
||||
# The positional-update corpus
|
||||
|
||||
MongoDB has three ways to say "descend into this array" in an update path:
|
||||
|
||||
| | since | pinned crud corpus |
|
||||
|---|---|---|
|
||||
| `$[<identifier>]` with `arrayFilters` | 3.6 | 4 files, 14 cases |
|
||||
| `$[]` all-positional | 3.6 | **nothing** |
|
||||
| `$` positional | ancient | **nothing** |
|
||||
|
||||
All three shared one code path in this engine, and that path overwrote the
|
||||
array it was supposed to walk — `y: [{b: 3}, {b: 1}]` became
|
||||
`y: {"$[i]": {"b": 2}}`, every element discarded, `ok: 1`. Only the
|
||||
`$[<identifier>]` third of it was externally visible, so a fix measured
|
||||
against the pinned corpus alone would have gone green while two of the three
|
||||
still destroyed data. That is `docs/M3_ARRAYFILTERS_DESIGN_REVIEW.md` §3, and
|
||||
this directory is its consequence.
|
||||
|
||||
## The one rule
|
||||
|
||||
**Inputs are authored here; expectations are measured against a real mongod.**
|
||||
|
||||
The same rule as `tests/spec/aggregate/`, for the same reason: a corpus we
|
||||
write end to end can encode our own bugs as expectations and will then agree
|
||||
with us forever. It is not a hypothetical risk here — the design review's own
|
||||
guesses about `$[]`, about missing paths and about upserts were all wrong
|
||||
before mongod was asked.
|
||||
|
||||
`sources/*.json` holds documents and operations and nothing else. `record.js`
|
||||
asks mongod 8.3.7 what each one answers.
|
||||
|
||||
## Running it
|
||||
|
||||
```sh
|
||||
node tests/spec/run.js --suite-dir tests/spec/positional
|
||||
```
|
||||
|
||||
The same runner as the crud corpus, pointed elsewhere — the entity model, the
|
||||
matchers, `outcome` verification and the skip accounting come for free.
|
||||
`--scorecard` is refused with `--suite-dir`: `tests/spec/scorecard.txt` is the
|
||||
crud corpus's record and the milestones are compared against it.
|
||||
|
||||
## Re-recording
|
||||
|
||||
```sh
|
||||
mongod --port 27099 --dbpath /tmp/mongo-corpus &
|
||||
node tests/spec/positional/record.js --mongod-port 27099
|
||||
```
|
||||
|
||||
Writes `<name>.json` for every `sources/<name>.json`. The generated files are
|
||||
committed: they *are* the corpus, and regenerating them is how a disagreement
|
||||
with mongod gets re-measured rather than argued about.
|
||||
|
||||
Three things to know when adding cases:
|
||||
|
||||
- **Every case records its `outcome`, including the refusals.** A refusal that
|
||||
left the document mangled looks identical to a clean one in `expectError`
|
||||
alone, and a mangled document is the entire reason this corpus exists.
|
||||
- **Errors record the code, not the message.** Message text is mongod's to
|
||||
change between releases. Several of these messages also embed a
|
||||
shell-syntax rendering of the offending element, which no formatter in this
|
||||
server produces.
|
||||
- **A case needing its own documents reseeds through operations**
|
||||
(`deleteMany` + `insertMany`), not `initialData`: the unified format's
|
||||
`initialData` is per file and the runner seeds it once per test. This keeps
|
||||
one file per operator rather than one file per document shape.
|
||||
|
||||
Every construct here is 3.6 or older, so nothing depends on a server newer
|
||||
than the 4.4 this one reports. A case that did should be left out rather than
|
||||
annotated — recording it from mongod 8.x and judging it against a 4.4 answer
|
||||
measures the version gap, not the engine.
|
||||
|
||||
## Where it stands
|
||||
|
||||
Recorded against mongod 8.3.7, run against the server at the positional
|
||||
refusal:
|
||||
|
||||
```
|
||||
filtered.json 8 pass 19 fail 0 skip
|
||||
all-positional.json 3 pass 9 fail 0 skip
|
||||
first-positional.json 4 pass 8 fail 0 skip
|
||||
```
|
||||
|
||||
Red by construction and by design. The 15 that pass are the refusals where
|
||||
this server's code already matches mongod's; of the 36 that fail, 31 are the
|
||||
constructs themselves answering "not implemented by this server", and 5 are
|
||||
refusals whose code differs — four of them the `arrayFilters` validation this
|
||||
server cannot do yet because it never parses the option.
|
||||
|
||||
That is the intended shape. `tests/spec/aggregate/expressions.json` was
|
||||
recorded at 1 pass / 26 fail before the evaluator existed and is green now;
|
||||
this is the same gate at the same stage.
|
||||
|
||||
## What recording it settled
|
||||
|
||||
None of this is guessable, and the first row contradicts what the review
|
||||
assumed:
|
||||
|
||||
| | mongod |
|
||||
|---|---|
|
||||
| `y.$[i].c.$[i].d`, one identifier reused at two levels | **accepted** — not a duplicate-identifier error |
|
||||
| `$[]` over an empty array | no-op, `modifiedCount: 0` |
|
||||
| `$[]` where one element is not a document | error **28**, where the other path failures are 2 |
|
||||
| `$[]` or `$[i]` where the path is missing, or is not an array | error 2 — a positional segment never *creates* |
|
||||
| an upsert that would insert | error 2; upsert gets no special case |
|
||||
| `$` when the query never touched the array | error 2 |
|
||||
| `$` where several elements match | only the first is written |
|
||||
| `$`, `$[]` or `$[i]` in first position | error 2 |
|
||||
| `$` twice in one path | error 2 |
|
||||
| `arrayFilters` alongside a replacement | ignored, no error |
|
||||
| an array filter the update never uses | error 9 |
|
||||
| a filter with no top-level field, two of them, or a duplicate identifier | error 9 |
|
||||
|
||||
The 2-versus-9 split is *nearly* "9 is the `arrayFilters` array judged on its
|
||||
own, 2 is anything needing the update or the stored document" — and it is not
|
||||
a clean rule: an identifier that is not a lowercase alphanumeric name carries
|
||||
the same `Error parsing array filter` prefix as the 9s and is a 2. Recorded as
|
||||
measured rather than tidied, because deriving the codes from the rule would
|
||||
get that row wrong.
|
||||
Reference in New Issue
Block a user