tests/spec: an aggregation corpus, recorded from mongod
M2.5's gate, built before the milestone it gates -- the same order that put
`expectEvents` before the free list in M1 and Tier 0 before everything in M2.
`mongodb/specifications` has no aggregation suite, which is amendment A6's
central finding, so this milestone has to bring its own. The hazard in a corpus
we author is obvious and fatal: it can encode our own bugs as expectations and
then agree with us forever. So the split is enforced by the tooling.
`sources/*.json` holds documents and pipelines and nothing else; `record.js`
asks a real mongod 8.3.7 what each pipeline answers and writes the unified-
format file from the reply. Inputs authored, expectations measured -- the
discipline that corrected three assumptions in M1's session work and every
error code in M2, where the alternative would have shipped both times.
No second runner. `run.js --suite-dir` points the existing one somewhere else,
so the entity model, the matchers, the skip accounting and `expectEvents` come
for free; a second runner would drift from the first exactly where it mattered.
`--scorecard` is refused with `--suite-dir`, because `scorecard.txt` is the crud
corpus's record and the milestones are compared against it -- writing it from an
unrelated run would replace that record silently.
Errors record the code and not the message: message text is mongod's to change
between releases. Group pipelines end in a `$sort`, because group output order
is unspecified and a case depending on it would fail for the wrong reason on
either server.
The first source covers `$group`: nine accumulators including the edge cases
that decide an implementation -- `$avg` over a group whose values are not
numbers, `$min` of a field no document has, `$push` skipping a missing field,
`$first`/`$last` against input order, grouping on an array, a compound `_id`.
Where it starts, run against the M2 tip:
group-accumulators.json 9 pass 10 fail 0 skip
The nine include the four refusals M2 added, which answer with mongod's own
codes -- so the corpus already confirms that half. The ten are the milestone.
The crud corpus is unchanged at 201/90/196.
This commit is contained in:
69
tests/spec/aggregate/README.md
Normal file
69
tests/spec/aggregate/README.md
Normal file
@@ -0,0 +1,69 @@
|
||||
# The aggregation corpus
|
||||
|
||||
`mongodb/specifications` has no aggregation suite. The thirteen
|
||||
`aggregate-*.json` files this project runs come from `crud` and test the
|
||||
aggregate *command* — cursors, read concern, the write stages, collation,
|
||||
`let`. They touch stages barely and expressions not at all: `$lookup`,
|
||||
`$unwind`, `$facet`, `$addFields` and `$replaceRoot` appear nowhere in the
|
||||
pinned corpus. That is PLAN amendment A6, and this directory is its
|
||||
consequence: M2.5 has to bring its own gate.
|
||||
|
||||
## The one rule
|
||||
|
||||
**Inputs are authored here; expectations are measured against a real mongod.**
|
||||
|
||||
A corpus we write is a corpus that can encode our own bugs as expectations, and
|
||||
it would then agree with us forever. So `sources/*.json` holds documents and
|
||||
pipelines and nothing else, and `record.js` asks mongod 8.3.7 what each pipeline
|
||||
answers. It is the same discipline that corrected three assumptions in M1's
|
||||
session work and every error code in M2 — the alternative, in both cases, would
|
||||
have shipped.
|
||||
|
||||
## Running it
|
||||
|
||||
```sh
|
||||
node tests/spec/run.js --suite-dir tests/spec/aggregate
|
||||
```
|
||||
|
||||
The same runner as the crud corpus, pointed elsewhere. Sharing it is the point:
|
||||
the entity model, the matchers, the skip accounting and `expectEvents` come for
|
||||
free, and a second runner would drift from the first exactly where it mattered.
|
||||
|
||||
`--scorecard` is refused with `--suite-dir`: `tests/spec/scorecard.txt` is the
|
||||
crud corpus's record and the milestones are compared against it.
|
||||
|
||||
## Re-recording
|
||||
|
||||
```sh
|
||||
mongod --port 27099 --dbpath /tmp/mongo-corpus &
|
||||
node tests/spec/aggregate/record.js --mongod-port 27099
|
||||
```
|
||||
|
||||
Writes `<name>.json` for every `sources/<name>.json`. The generated files are
|
||||
committed: they *are* the corpus, and regenerating them is how a disagreement
|
||||
with mongod gets re-measured rather than argued about.
|
||||
|
||||
Two things to know when adding cases:
|
||||
|
||||
- **End a `$group` pipeline with a `$sort`.** Group output order is unspecified,
|
||||
and a case that depended on it would fail for the wrong reason on either
|
||||
server.
|
||||
- **Errors record the code, not the message.** Message text is mongod's to
|
||||
change between releases; a corpus that pinned it would break for the wrong
|
||||
reason.
|
||||
|
||||
Leave out any case whose answer depends on a server newer than the 4.4 this
|
||||
server reports — recording it from mongod 8.x and judging it against a 4.4
|
||||
answer measures the version gap, not the engine.
|
||||
|
||||
## Where it stands
|
||||
|
||||
Recorded against mongod 8.3.7, run against the M2 tip:
|
||||
|
||||
```
|
||||
group-accumulators.json 9 pass 10 fail 0 skip
|
||||
```
|
||||
|
||||
The nine include the four refusals M2 added, which answer with mongod's own
|
||||
codes. The ten are M2.5's work: `$avg`, `$min`, `$max`, `$first`, `$last`,
|
||||
`$push`, `$addToSet`, `$count`, and a compound `_id`.
|
||||
Reference in New Issue
Block a user