diff --git a/tests/spec/README.md b/tests/spec/README.md index f2905bf..3f4cc85 100644 --- a/tests/spec/README.md +++ b/tests/spec/README.md @@ -41,29 +41,43 @@ result is a real pass: Supported: `client`/`database`/`collection` entities, `initialData`, `outcome`, `expectError` (code, codeName, contains, labels, errorResponse), -`saveResultAsEntity`, `runOnRequirements` gating, and the `$$type`, -`$$exists`, `$$unsetOrMatches`, `$$matchesEntity`, `$$matchesHexBytes` -operators. +`expectEvents`, `saveResultAsEntity`, `runOnRequirements` gating, and the +`$$type`, `$$exists`, `$$unsetOrMatches`, `$$matchesEntity`, +`$$matchesHexBytes` operators. -**Not asserted yet: `expectEvents`** (command monitoring). Those assertions are -about the command shape the driver emits rather than result semantics. Ignoring -them lets some cases pass that a complete runner would fail, so **treat `pass` -as an upper bound** until M1 wires events up. This is stated again at the top of -`scorecard.txt` so the number is never read out of context. +`expectEvents` compares the commands the driver actually sent against the +expectation, **exact in number and in order**, with `command` and `reply` +matched as root documents so the driver's own additions (`lsid`, `$db`) are +allowed. It is what makes a pass mean the engine answered correctly *and* was +asked the right question — 354 of the 487 cases declare events, and before this +was asserted a case could send the wrong command and still be counted a pass. +**Scorecards recorded before it landed are not comparable**; there, `pass` was +an upper bound by construction. + +Still unasserted within events, each reported as SKIP at the point of +assertion: `cmap` and `sdam` event types, `ignoreExtraEvents`, `hasServiceId`, +`hasServerConnectionId`, and `maxTimeMS` in an expected command — the runner +puts CSOT `timeoutMS` on every client, and CSOT overwrites `maxTimeMS` with +what is left of that budget, so the value on the wire is the harness's. That is +the only assertion this runner declines to make; one case in the corpus is +affected. Not supported, each reported as SKIP with a reason and never as PASS: session and bucket entities (M4 / GridFS), `failPoint`, client-side encryption, `testRunner` operations, and any operation or matcher the runner does not know. +`MFDB_DUMP_EVENTS=1` prints each case's observed command stream, which is the +fastest way to tell a wrong answer from a command the driver never sent. + ## Reading the scorecard `scorecard.txt` records the totals, a per-file breakdown, and every non-passing case with its reason. The distinction that matters: - **FAIL** — the engine answered, and answered differently from the spec. Real - work. An operation that never answered inside `--op-timeout-ms` (default 3 s, - enforced by the driver itself via CSOT `timeoutMS`) is also a FAIL, because - "no answer" is a result. There is a second, much longer `--case-timeout-ms` + work. An operation that never answered inside `--op-timeout-ms` (default + 10 s, enforced by the driver itself via CSOT `timeoutMS`) is also a FAIL, + because "no answer" is a result. There is a second, much longer `--case-timeout-ms` backstop for a hang the driver cannot see; if it ever fires, treat the run with suspicion — see the trap below. - **SKIP** — nobody claims anything. Either the suite needs a feature whose diff --git a/tests/spec/run.js b/tests/spec/run.js index 021b193..a630bd9 100644 --- a/tests/spec/run.js +++ b/tests/spec/run.js @@ -31,8 +31,9 @@ const SUITE_DIR = path.join(__dirname, 'specifications', 'source', 'crud', 'test const SCORECARD = path.join(__dirname, 'scorecard.txt'); const REPO = path.join(__dirname, '..', '..'); -// The runner implements schema 1.0-1.9 features that CRUD tests actually use. -// A file declaring more than this is skipped whole rather than half-run. +// The runner implements the schema features that CRUD tests actually use, up +// to this version. A file declaring more than this is skipped whole rather +// than half-run. const MAX_SCHEMA = [1, 24]; const argv = process.argv.slice(2); @@ -897,10 +898,15 @@ function scorecardText(results, tot, errored, server, nfiles) { L.push('# - the suite needs a feature whose milestone has not landed (sessions M4,'); L.push('# auth M7, failPoints, gridfs) -- counted as skip, never as pass;'); L.push('# - or the runner itself does not implement the operation/matcher yet.'); - L.push('# Deliberately NOT asserted yet: expectEvents (command monitoring). Those'); - L.push('# assertions are about driver-visible command shape rather than result'); - L.push('# semantics; ignoring them makes some cases pass that a full runner would'); - L.push('# fail, so treat `pass` as an upper bound until M1 wires events up.'); + L.push('# expectEvents IS asserted: the commands the driver sent are compared to the'); + L.push('# expectation exactly, in number and in order, with `command` and `reply`'); + L.push('# matched as root documents. A pass therefore means the engine answered'); + L.push('# correctly *and* was asked the right question. Not comparable to any'); + L.push('# scorecard recorded before that landed, where `pass` was an upper bound.'); + L.push('# Still unasserted, each reported as skip where it is asserted, never as'); + L.push('# pass: cmap and sdam events, ignoreExtraEvents, hasServiceId,'); + L.push('# hasServerConnectionId, and maxTimeMS in an expected command (CSOT rewrites'); + L.push('# it -- the only assertion this runner declines to make).'); L.push(''); L.push(`total\t${tot.pass} pass\t${tot.fail} fail\t${tot.skip} skip\t${nfiles} files\t${errored} errored`); L.push(''); diff --git a/tests/spec/scorecard.txt b/tests/spec/scorecard.txt index 28b048a..5556ea8 100644 --- a/tests/spec/scorecard.txt +++ b/tests/spec/scorecard.txt @@ -8,10 +8,15 @@ # - the suite needs a feature whose milestone has not landed (sessions M4, # auth M7, failPoints, gridfs) -- counted as skip, never as pass; # - or the runner itself does not implement the operation/matcher yet. -# Deliberately NOT asserted yet: expectEvents (command monitoring). Those -# assertions are about driver-visible command shape rather than result -# semantics; ignoring them makes some cases pass that a full runner would -# fail, so treat `pass` as an upper bound until M1 wires events up. +# expectEvents IS asserted: the commands the driver sent are compared to the +# expectation exactly, in number and in order, with `command` and `reply` +# matched as root documents. A pass therefore means the engine answered +# correctly *and* was asked the right question. Not comparable to any +# scorecard recorded before that landed, where `pass` was an upper bound. +# Still unasserted, each reported as skip where it is asserted, never as +# pass: cmap and sdam events, ignoreExtraEvents, hasServiceId, +# hasServerConnectionId, and maxTimeMS in an expected command (CSOT rewrites +# it -- the only assertion this runner declines to make). total 159 pass 132 fail 196 skip 175 files 0 errored