tests/spec/README.md

@@ -41,29 +41,43 @@ result is a real pass:
 
 Supported: `client`/`database`/`collection` entities, `initialData`,
 `outcome`, `expectError` (code, codeName, contains, labels, errorResponse),
-`saveResultAsEntity`, `runOnRequirements` gating, and the `$$type`,
+`expectEvents`, `saveResultAsEntity`, `runOnRequirements` gating, and the
-`$$exists`, `$$unsetOrMatches`, `$$matchesEntity`, `$$matchesHexBytes`
+`$$type`, `$$exists`, `$$unsetOrMatches`, `$$matchesEntity`,
-operators.
+`$$matchesHexBytes` operators.
 
-**Not asserted yet: `expectEvents`** (command monitoring). Those assertions are
+`expectEvents` compares the commands the driver actually sent against the
-about the command shape the driver emits rather than result semantics. Ignoring
+expectation, **exact in number and in order**, with `command` and `reply`
-them lets some cases pass that a complete runner would fail, so **treat `pass`
+matched as root documents so the driver's own additions (`lsid`, `$db`) are
-as an upper bound** until M1 wires events up. This is stated again at the top of
+allowed. It is what makes a pass mean the engine answered correctly *and* was
-`scorecard.txt` so the number is never read out of context.
+asked the right question — 354 of the 487 cases declare events, and before this
+was asserted a case could send the wrong command and still be counted a pass.
+**Scorecards recorded before it landed are not comparable**; there, `pass` was
+an upper bound by construction.
+
+Still unasserted within events, each reported as SKIP at the point of
+assertion: `cmap` and `sdam` event types, `ignoreExtraEvents`, `hasServiceId`,
+`hasServerConnectionId`, and `maxTimeMS` in an expected command — the runner
+puts CSOT `timeoutMS` on every client, and CSOT overwrites `maxTimeMS` with
+what is left of that budget, so the value on the wire is the harness's. That is
+the only assertion this runner declines to make; one case in the corpus is
+affected.
 
 Not supported, each reported as SKIP with a reason and never as PASS: session
 and bucket entities (M4 / GridFS), `failPoint`, client-side encryption,
 `testRunner` operations, and any operation or matcher the runner does not know.
 
+`MFDB_DUMP_EVENTS=1` prints each case's observed command stream, which is the
+fastest way to tell a wrong answer from a command the driver never sent.
+
 ## Reading the scorecard
 
 `scorecard.txt` records the totals, a per-file breakdown, and every
 non-passing case with its reason. The distinction that matters:
 
 - **FAIL** — the engine answered, and answered differently from the spec. Real
-  work. An operation that never answered inside `--op-timeout-ms` (default 3 s,
+  work. An operation that never answered inside `--op-timeout-ms` (default
-  enforced by the driver itself via CSOT `timeoutMS`) is also a FAIL, because
+  10 s, enforced by the driver itself via CSOT `timeoutMS`) is also a FAIL,
-  "no answer" is a result. There is a second, much longer `--case-timeout-ms`
+  because "no answer" is a result. There is a second, much longer `--case-timeout-ms`
   backstop for a hang the driver cannot see; if it ever fires, treat the run
   with suspicion — see the trap below.
 - **SKIP** — nobody claims anything. Either the suite needs a feature whose

tests/spec/run.js

@@ -31,8 +31,9 @@ const SUITE_DIR = path.join(__dirname, 'specifications', 'source', 'crud', 'test
 const SCORECARD = path.join(__dirname, 'scorecard.txt');
 const REPO = path.join(__dirname, '..', '..');
 
-// The runner implements schema 1.0-1.9 features that CRUD tests actually use.
+// The runner implements the schema features that CRUD tests actually use, up
-// A file declaring more than this is skipped whole rather than half-run.
+// to this version. A file declaring more than this is skipped whole rather
+// than half-run.
 const MAX_SCHEMA = [1, 24];
 
 const argv = process.argv.slice(2);
@@ -897,10 +898,15 @@ function scorecardText(results, tot, errored, server, nfiles) {
     L.push('#   - the suite needs a feature whose milestone has not landed (sessions M4,');
     L.push('#     auth M7, failPoints, gridfs) -- counted as skip, never as pass;');
     L.push('#   - or the runner itself does not implement the operation/matcher yet.');
-    L.push('# Deliberately NOT asserted yet: expectEvents (command monitoring). Those');
+    L.push('# expectEvents IS asserted: the commands the driver sent are compared to the');
-    L.push('# assertions are about driver-visible command shape rather than result');
+    L.push('# expectation exactly, in number and in order, with `command` and `reply`');
-    L.push('# semantics; ignoring them makes some cases pass that a full runner would');
+    L.push('# matched as root documents. A pass therefore means the engine answered');
-    L.push('# fail, so treat `pass` as an upper bound until M1 wires events up.');
+    L.push('# correctly *and* was asked the right question. Not comparable to any');
+    L.push('# scorecard recorded before that landed, where `pass` was an upper bound.');
+    L.push('# Still unasserted, each reported as skip where it is asserted, never as');
+    L.push('# pass: cmap and sdam events, ignoreExtraEvents, hasServiceId,');
+    L.push('# hasServerConnectionId, and maxTimeMS in an expected command (CSOT rewrites');
+    L.push('# it -- the only assertion this runner declines to make).');
     L.push('');
     L.push(`total\t${tot.pass} pass\t${tot.fail} fail\t${tot.skip} skip\t${nfiles} files\t${errored} errored`);
     L.push('');

tests/spec/scorecard.txt

@@ -8,10 +8,15 @@
 #   - the suite needs a feature whose milestone has not landed (sessions M4,
 #     auth M7, failPoints, gridfs) -- counted as skip, never as pass;
 #   - or the runner itself does not implement the operation/matcher yet.
-# Deliberately NOT asserted yet: expectEvents (command monitoring). Those
+# expectEvents IS asserted: the commands the driver sent are compared to the
-# assertions are about driver-visible command shape rather than result
+# expectation exactly, in number and in order, with `command` and `reply`
-# semantics; ignoring them makes some cases pass that a full runner would
+# matched as root documents. A pass therefore means the engine answered
-# fail, so treat `pass` as an upper bound until M1 wires events up.
+# correctly *and* was asked the right question. Not comparable to any
+# scorecard recorded before that landed, where `pass` was an upper bound.
+# Still unasserted, each reported as skip where it is asserted, never as
+# pass: cmap and sdam events, ignoreExtraEvents, hasServiceId,
+# hasServerConnectionId, and maxTimeMS in an expected command (CSOT rewrites
+# it -- the only assertion this runner declines to make).
 
 total	159 pass	132 fail	196 skip	175 files	0 errored