bb8cdd964b
The disclaimer was accurate for as long as it stood -- events were not read,
so `pass` was an upper bound and saying otherwise would have been a lie about
the number. It is now a lie in the other direction, so it goes, replaced by
what is actually true: events are compared exactly, in number and in order,
which is what makes a pass mean the engine answered correctly *and* was asked
the right question. The header says plainly that scorecards recorded before
this are not comparable, and enumerates what is still skipped inside events
rather than leaving "asserted" to be read as "asserted completely".
Two facts in the docs had gone stale and are corrected here because this is
the commit that rereads them:
- README said `--op-timeout-ms` defaults to 3 s. It has been 10 s since the
commit that explains, at length and directly above the constant, why 3 s
was wrong. A stale number in exactly the place that warns against
tightening it is worse than no number.
- `MAX_SCHEMA` is [1, 24]; the comment above it still claimed 1.0-1.9.
Totals unchanged at 159/132/196 -- this commit only rewrites prose, and the
scorecard is re-recorded so its header matches the runner that produced it.
151 lines
7.5 KiB
Markdown
151 lines
7.5 KiB
Markdown
# MongoDB spec tests
|
|
|
|
PLAN D2 makes the official MongoDB JSON specification suites the gate for
|
|
command semantics: it turns "maximally compatible" into a concrete list of test
|
|
files rather than a judgement call. This directory holds the runner and the
|
|
committed scorecard.
|
|
|
|
```sh
|
|
bash tests/spec/fetch.sh # pinned suites (~175 files, gitignored)
|
|
zig build # the runner spawns this binary
|
|
node tests/spec/run.js # run everything
|
|
node tests/spec/run.js --scorecard # ... and rewrite scorecard.txt
|
|
node tests/spec/run.js --file find.json --verbose
|
|
node tests/spec/run.js --url mongodb://127.0.0.1:27020 # use a server you started
|
|
```
|
|
|
|
## What is pinned, and why both halves matter
|
|
|
|
- **Suites**: `mongodb/specifications` @ `615e0f9`, in `fetch.sh`.
|
|
- **Driver**: `mongodb@7.5.0`, via `tests/e2e/package-lock.json`.
|
|
|
|
A scorecard is only comparable across milestones if both are pinned — otherwise
|
|
a delta could be an upstream test change rather than an engine change. Bump
|
|
either one in its own commit and re-record the scorecard in that same commit.
|
|
|
|
The suites are fetched rather than vendored: they are someone else's corpus,
|
|
upstream rewrites them wholesale, and a pinned commit gives the same
|
|
reproducibility without putting them in this repo's history.
|
|
|
|
## Scope
|
|
|
|
`source/crud/tests/unified/` — 175 files. The aggregate tests live there too
|
|
(`aggregate*.json`), so this one directory is PLAN M0's "crud + aggregate".
|
|
|
|
The runner implements the unified test format's **Evaluating Matches**
|
|
algorithm as written in the spec, including the two rules that decide whether a
|
|
result is a real pass:
|
|
|
|
- extra keys in the actual document are tolerated **only** in a root document;
|
|
- numeric types (int32 / int64 / double) compare flexibly.
|
|
|
|
Supported: `client`/`database`/`collection` entities, `initialData`,
|
|
`outcome`, `expectError` (code, codeName, contains, labels, errorResponse),
|
|
`expectEvents`, `saveResultAsEntity`, `runOnRequirements` gating, and the
|
|
`$$type`, `$$exists`, `$$unsetOrMatches`, `$$matchesEntity`,
|
|
`$$matchesHexBytes` operators.
|
|
|
|
`expectEvents` compares the commands the driver actually sent against the
|
|
expectation, **exact in number and in order**, with `command` and `reply`
|
|
matched as root documents so the driver's own additions (`lsid`, `$db`) are
|
|
allowed. It is what makes a pass mean the engine answered correctly *and* was
|
|
asked the right question — 354 of the 487 cases declare events, and before this
|
|
was asserted a case could send the wrong command and still be counted a pass.
|
|
**Scorecards recorded before it landed are not comparable**; there, `pass` was
|
|
an upper bound by construction.
|
|
|
|
Still unasserted within events, each reported as SKIP at the point of
|
|
assertion: `cmap` and `sdam` event types, `ignoreExtraEvents`, `hasServiceId`,
|
|
`hasServerConnectionId`, and `maxTimeMS` in an expected command — the runner
|
|
puts CSOT `timeoutMS` on every client, and CSOT overwrites `maxTimeMS` with
|
|
what is left of that budget, so the value on the wire is the harness's. That is
|
|
the only assertion this runner declines to make; one case in the corpus is
|
|
affected.
|
|
|
|
Not supported, each reported as SKIP with a reason and never as PASS: session
|
|
and bucket entities (M4 / GridFS), `failPoint`, client-side encryption,
|
|
`testRunner` operations, and any operation or matcher the runner does not know.
|
|
|
|
`MFDB_DUMP_EVENTS=1` prints each case's observed command stream, which is the
|
|
fastest way to tell a wrong answer from a command the driver never sent.
|
|
|
|
## Reading the scorecard
|
|
|
|
`scorecard.txt` records the totals, a per-file breakdown, and every
|
|
non-passing case with its reason. The distinction that matters:
|
|
|
|
- **FAIL** — the engine answered, and answered differently from the spec. Real
|
|
work. An operation that never answered inside `--op-timeout-ms` (default
|
|
10 s, enforced by the driver itself via CSOT `timeoutMS`) is also a FAIL,
|
|
because "no answer" is a result. There is a second, much longer `--case-timeout-ms`
|
|
backstop for a hang the driver cannot see; if it ever fires, treat the run
|
|
with suspicion — see the trap below.
|
|
- **SKIP** — nobody claims anything. Either the suite needs a feature whose
|
|
milestone has not landed, or the runner does not implement it yet.
|
|
|
|
M0's gate (PLAN D7.6) is only that the harness exists and the baseline is
|
|
recorded. **A red baseline is the expected state**, so `run.js` exits 0 as long
|
|
as it ran; it is a measuring tool, not a pass/fail gate. Later milestones move
|
|
the numbers, and each one commits the new scorecard (PLAN D9).
|
|
|
|
## A trap worth knowing about: the harness can invent failures
|
|
|
|
The first baseline attempt reported ~77 timeout FAILs that did not exist. Every
|
|
case from one point onward timed out, while a `ping` from a separate process
|
|
answered instantly — which read convincingly as a server-side wedge, and was
|
|
not.
|
|
|
|
The cause was in this runner. `buildEntities` opened `MongoClient`s, and a case
|
|
that timed out before it returned left them unclosed; each one keeps a
|
|
connection pool and a heartbeat timer. Once enough accumulated, Node's event
|
|
loop was starved badly enough that the per-case timer fired before operations
|
|
could finish. Then every later case "failed".
|
|
|
|
Two things guard it now: per-test clients are owned by the caller and closed
|
|
unconditionally, including on a partial failure; and the run ends by checking
|
|
how many timers are still active, warning loudly if the answer is more than a
|
|
handful.
|
|
|
|
The general rule, since it will come up again: **a run with a long unbroken tail
|
|
of timeouts is a harness bug until proven otherwise.** Confirm it by running the
|
|
first timing-out file on its own — if it passes in isolation, the failures are
|
|
this runner's, not the engine's.
|
|
|
|
### ... but the third time it was the engine
|
|
|
|
A later attempt produced 166 timeout FAILs starting at file 70. I first blamed
|
|
machine load — a concurrent `zig build test` against a then-3-second budget —
|
|
and that was **wrong**. The evidence against it: the collapse reproduced on an
|
|
idle machine, at the same file, with a 10 s budget.
|
|
|
|
The actual cause was a leaked catalog lock in the engine, and it is worth
|
|
knowing how it hid. `db-aggregate.json` sends `{aggregate: 1}`, which names no
|
|
collection; dispatch resolved the namespace after taking the catalog lock and
|
|
bailed with a plain `return`, holding it shared forever. A leaked *shared* lock
|
|
is invisible to readers, so the server stayed perfectly responsive — an external
|
|
prober got `ok 15ms` right through the hang — and only the next write that had
|
|
to take the catalog exclusive to create a collection blocked. The failure
|
|
therefore surfaced one file later, on a different connection, as a client-side
|
|
timeout with nothing pointing at its cause.
|
|
|
|
Two lessons for using this runner:
|
|
|
|
- **A healthy-looking server does not exonerate the engine.** Probe with the
|
|
operation that is actually stuck, not with `ping`.
|
|
- **The driver's own command log is the fastest way in.** It showed an insert
|
|
sitting for exactly `socketTimeoutMS` against an idle engine, which is what
|
|
turned a week-long-looking mystery into a five-line fix:
|
|
|
|
```sh
|
|
MONGODB_LOG_COMMAND=debug MONGODB_LOG_PATH=stderr \
|
|
node tests/spec/run.js --skip 68 --limit 2 2>drv.log
|
|
```
|
|
|
|
`--skip`/`--limit` exist for exactly this: the collapse reduced to a
|
|
reproducible two-file window, which is what made it tractable.
|
|
|
|
Still worth recording the baseline on an otherwise idle machine, and do not
|
|
tighten `--op-timeout-ms` to make a run finish sooner — a tight budget turns
|
|
load into apparent engine failures, which is how I misdiagnosed this once
|
|
already.
|