From 90de7820da9274c0dbf7db4836fbf90b87f59494 Mon Sep 17 00:00:00 2001 From: Aleksey Shakhmatov Date: Mon, 3 Aug 2026 18:58:01 +0300 Subject: [PATCH] tests/spec: MongoDB spec-test runner and the M0 scorecard PLAN D2 makes the official specification suites the gate for command semantics; D7.6 asks for the harness to exist at M0 with a recorded baseline. This is that harness, pinned on both sides -- mongodb/specifications @ 615e0f9 and mongodb@7.5.0 -- because a scorecard is only comparable across milestones if a delta cannot be an upstream test change. It implements the unified format's Evaluating Matches algorithm as written, including the two rules that decide whether a pass is earned: extra keys are tolerated only in a root document, and numeric types compare flexibly. Anything unimplemented is a SKIP with a reason, never a pass, and the one assertion class not yet checked -- expectEvents, i.e. command monitoring -- is disclosed at the top of the scorecard so `pass` reads as an upper bound. First honest run: 131 pass, 161 fail, 195 skip over 175 files, zero timeouts. Getting there took four attempts, and the failures are documented in the README because each would have shipped a scorecard claiming a compatibility gap that did not exist. Two were genuine leaks in this runner (clients left open when a case timed out; clients registered for cleanup only after `await connect()`, plus abandoned cases still creating more). The third I misdiagnosed as machine load. The fourth attempt found the real cause: a leaked catalog lock in the engine, fixed separately, which alone accounts for the jump from 45 passes to 131. So the runner carries its own guards: per-operation CSOT timeouts so work is never abandoned, an active-handle census per file, an end-of-run tripwire for stray timers, a hard stop if the server dies rather than emitting hundreds of misleading ECONNREFUSED failures, and --skip/--limit for bisecting a run whose failures depend on position. The README states the rule plainly -- a long unbroken tail of timeouts is a harness bug until proven otherwise -- and the two commands that settle it. Also fixes bench-run.sh, which copied its report over bench-latest.txt unconditionally, including after a run that only warned -- so a degraded run could silently replace the baseline that PLAN D7.5 makes a milestone gate. --- .gitignore | 2 + PLAN.md | 52 ++- tests/e2e/bench-run.sh | 13 + tests/spec/README.md | 136 +++++++ tests/spec/fetch.sh | 44 +++ tests/spec/run.js | 743 +++++++++++++++++++++++++++++++++++++++ tests/spec/scorecard.txt | 503 ++++++++++++++++++++++++++ 7 files changed, 1487 insertions(+), 6 deletions(-) create mode 100644 tests/spec/README.md create mode 100755 tests/spec/fetch.sh create mode 100644 tests/spec/run.js create mode 100644 tests/spec/scorecard.txt diff --git a/.gitignore b/.gitignore index 0b04fc7..b398b00 100644 --- a/.gitignore +++ b/.gitignore @@ -2,3 +2,5 @@ zig-out/ *.log node_modules/ +# Pinned upstream spec suites, fetched by tests/spec/fetch.sh (PLAN D2). +tests/spec/specifications/ diff --git a/PLAN.md b/PLAN.md index 243665c..a3aa62a 100644 --- a/PLAN.md +++ b/PLAN.md @@ -380,11 +380,36 @@ stay in the arena — it outlives the command; it is only the ArrayList's own buffer whose allocator has to match its `deinit`). Add an e2e case for a bare `$sort` pipeline, and mutation-check it by restoring `arena` on the append. +### Bug found by the spec harness: a nameless command leaked the catalog lock + +**Fixed. Severity: permanent denial of service, remotely triggerable by an +ordinary query.** This one blocked the M0 scorecard outright and cost three +invalid baselines. + +`dispatch` resolved the namespace *after* taking the catalog lock and bailed +with `orelse return` when the collection name was missing. A plain return runs +neither the `errdefer` nor the explicit unlocks, so the lock was held — shared — +for the life of the process. `db.aggregate(...)` reaches it: that sends +`{aggregate: 1}`, whose value is not a string. + +Worth understanding how it hid, because the shape recurs: a leaked *shared* lock +is invisible to readers. `ping` and `listDatabases` kept answering in +microseconds and an external prober saw `ok 15ms` straight through the hang, so +the server looked healthy. Only a write needing the catalog exclusive to create +a collection blocked — so the damage appeared one command later, on a different +connection, as a client-side timeout with nothing linking it to the cause. + +What actually found it: the driver's own command log, showing an insert sitting +for exactly `socketTimeoutMS` against an idle engine. + +Namespace resolution now happens before any lock is taken. Two lessons kept in +`tests/spec/README.md`: a responsive server does not exonerate the engine, and +probe with the operation that is stuck rather than with `ping`. + ### Bug found by the spec harness: unacknowledged writes corrupt the connection -Recorded here rather than fixed in M0, since it is M1's surface — but it is a -correctness bug, not a missing feature, and it is worth doing early because it -is a handful of lines. +**Fixed** (in M0 rather than M1 as originally recorded — it is a correctness +bug, not a missing feature, and it is a handful of lines). `wire.Message.flags` is parsed and stored but **never read**. A driver sending an unacknowledged write (`writeConcern: {w: 0}`) sets `moreToCome` (bit 0x2) @@ -403,9 +428,24 @@ next command on same connection FAILED: MongoUnexpectedServerResponseError: performance choice — breaks a connection on first use, and it is invisible to the existing e2e suites because none of them use it. -Fix: when `flags & 0x2` is set on an OP_MSG request, run the command and write -no reply. Add an e2e case for it (unacknowledged write, then a read on the same -connection), and mutation-check it by clearing the flag test. +Fixed by suppressing the reply when `flags & 0x2` is set on an OP_MSG request: +the command still runs. The e2e case pins `maxPoolSize` to 1, since with a +larger pool the driver may hand the next operation a different connection and +hide it. + +### What the harness was worth + +Its first honest run — after the two bugs above and two genuine leaks in the +runner itself — reports **131 pass, 161 fail, 195 skip** over 175 files, with +zero timeouts. Before the catalog-lock fix the same suite reported 45 passes: +that gap is the measure of what one leaked lock was hiding, and of why a +scorecard is only worth committing once it disagrees with nothing that passes in +isolation. + +The remaining failures are real work, and they cluster usefully: update-operator +gaps (`bad update`, `update must be a document` — M3), unimplemented commands +(`distinct`, `$merge`, `$out` — M2/M3), and result-shape mismatches in +`bulkWrite`/`insertMany`. That list, not the total, is the milestone backlog. --- diff --git a/tests/e2e/bench-run.sh b/tests/e2e/bench-run.sh index 964f1d6..707d906 100644 --- a/tests/e2e/bench-run.sh +++ b/tests/e2e/bench-run.sh @@ -57,6 +57,7 @@ for C in $CLIENTS; do MD_D=$(echo "$MD" | sed -E 's/.*\t([0-9.]+) docs\/s.*/\1/') if [ -z "$MFDB_D" ] || [ -z "$MD_D" ]; then echo "WARNING: concurrent run at $C clients produced no result (mfdb='$MFDB' md='$MD')" >&2 + DEGRADED=1 fi RATIO=$(node -e "const m=Number('$MFDB_D'),d=Number('$MD_D');console.log(d>0?(m/d).toFixed(1)+'x':'—')") echo "clients $C $MFDB_D $MD_D $RATIO" | tee -a "$CC" @@ -148,5 +149,17 @@ if [ -f "$LATEST" ]; then else echo "(no previous run to diff — this is the baseline)" fi +# bench-latest.txt is the baseline the next run diffs against, and PLAN D7.5 +# makes "no phase8 row regresses" a milestone gate -- so a run that only +# half-produced its numbers must not become the thing we compare to. This used +# to be an unconditional copy, which meant a degraded run silently replaced the +# baseline it was supposed to be measured against, and the missing rows then +# read as "no change" forever. +if [ "${DEGRADED:-0}" = "1" ]; then + echo >&2 + echo "NOT updating $LATEST: this run was degraded (see WARNING above)." >&2 + echo "The report is still at $REPORT if you want it." >&2 + exit 1 +fi cp "$REPORT" "$LATEST" echo; echo "latest: $LATEST" diff --git a/tests/spec/README.md b/tests/spec/README.md new file mode 100644 index 0000000..f2905bf --- /dev/null +++ b/tests/spec/README.md @@ -0,0 +1,136 @@ +# MongoDB spec tests + +PLAN D2 makes the official MongoDB JSON specification suites the gate for +command semantics: it turns "maximally compatible" into a concrete list of test +files rather than a judgement call. This directory holds the runner and the +committed scorecard. + +```sh +bash tests/spec/fetch.sh # pinned suites (~175 files, gitignored) +zig build # the runner spawns this binary +node tests/spec/run.js # run everything +node tests/spec/run.js --scorecard # ... and rewrite scorecard.txt +node tests/spec/run.js --file find.json --verbose +node tests/spec/run.js --url mongodb://127.0.0.1:27020 # use a server you started +``` + +## What is pinned, and why both halves matter + +- **Suites**: `mongodb/specifications` @ `615e0f9`, in `fetch.sh`. +- **Driver**: `mongodb@7.5.0`, via `tests/e2e/package-lock.json`. + +A scorecard is only comparable across milestones if both are pinned — otherwise +a delta could be an upstream test change rather than an engine change. Bump +either one in its own commit and re-record the scorecard in that same commit. + +The suites are fetched rather than vendored: they are someone else's corpus, +upstream rewrites them wholesale, and a pinned commit gives the same +reproducibility without putting them in this repo's history. + +## Scope + +`source/crud/tests/unified/` — 175 files. The aggregate tests live there too +(`aggregate*.json`), so this one directory is PLAN M0's "crud + aggregate". + +The runner implements the unified test format's **Evaluating Matches** +algorithm as written in the spec, including the two rules that decide whether a +result is a real pass: + +- extra keys in the actual document are tolerated **only** in a root document; +- numeric types (int32 / int64 / double) compare flexibly. + +Supported: `client`/`database`/`collection` entities, `initialData`, +`outcome`, `expectError` (code, codeName, contains, labels, errorResponse), +`saveResultAsEntity`, `runOnRequirements` gating, and the `$$type`, +`$$exists`, `$$unsetOrMatches`, `$$matchesEntity`, `$$matchesHexBytes` +operators. + +**Not asserted yet: `expectEvents`** (command monitoring). Those assertions are +about the command shape the driver emits rather than result semantics. Ignoring +them lets some cases pass that a complete runner would fail, so **treat `pass` +as an upper bound** until M1 wires events up. This is stated again at the top of +`scorecard.txt` so the number is never read out of context. + +Not supported, each reported as SKIP with a reason and never as PASS: session +and bucket entities (M4 / GridFS), `failPoint`, client-side encryption, +`testRunner` operations, and any operation or matcher the runner does not know. + +## Reading the scorecard + +`scorecard.txt` records the totals, a per-file breakdown, and every +non-passing case with its reason. The distinction that matters: + +- **FAIL** — the engine answered, and answered differently from the spec. Real + work. An operation that never answered inside `--op-timeout-ms` (default 3 s, + enforced by the driver itself via CSOT `timeoutMS`) is also a FAIL, because + "no answer" is a result. There is a second, much longer `--case-timeout-ms` + backstop for a hang the driver cannot see; if it ever fires, treat the run + with suspicion — see the trap below. +- **SKIP** — nobody claims anything. Either the suite needs a feature whose + milestone has not landed, or the runner does not implement it yet. + +M0's gate (PLAN D7.6) is only that the harness exists and the baseline is +recorded. **A red baseline is the expected state**, so `run.js` exits 0 as long +as it ran; it is a measuring tool, not a pass/fail gate. Later milestones move +the numbers, and each one commits the new scorecard (PLAN D9). + +## A trap worth knowing about: the harness can invent failures + +The first baseline attempt reported ~77 timeout FAILs that did not exist. Every +case from one point onward timed out, while a `ping` from a separate process +answered instantly — which read convincingly as a server-side wedge, and was +not. + +The cause was in this runner. `buildEntities` opened `MongoClient`s, and a case +that timed out before it returned left them unclosed; each one keeps a +connection pool and a heartbeat timer. Once enough accumulated, Node's event +loop was starved badly enough that the per-case timer fired before operations +could finish. Then every later case "failed". + +Two things guard it now: per-test clients are owned by the caller and closed +unconditionally, including on a partial failure; and the run ends by checking +how many timers are still active, warning loudly if the answer is more than a +handful. + +The general rule, since it will come up again: **a run with a long unbroken tail +of timeouts is a harness bug until proven otherwise.** Confirm it by running the +first timing-out file on its own — if it passes in isolation, the failures are +this runner's, not the engine's. + +### ... but the third time it was the engine + +A later attempt produced 166 timeout FAILs starting at file 70. I first blamed +machine load — a concurrent `zig build test` against a then-3-second budget — +and that was **wrong**. The evidence against it: the collapse reproduced on an +idle machine, at the same file, with a 10 s budget. + +The actual cause was a leaked catalog lock in the engine, and it is worth +knowing how it hid. `db-aggregate.json` sends `{aggregate: 1}`, which names no +collection; dispatch resolved the namespace after taking the catalog lock and +bailed with a plain `return`, holding it shared forever. A leaked *shared* lock +is invisible to readers, so the server stayed perfectly responsive — an external +prober got `ok 15ms` right through the hang — and only the next write that had +to take the catalog exclusive to create a collection blocked. The failure +therefore surfaced one file later, on a different connection, as a client-side +timeout with nothing pointing at its cause. + +Two lessons for using this runner: + +- **A healthy-looking server does not exonerate the engine.** Probe with the + operation that is actually stuck, not with `ping`. +- **The driver's own command log is the fastest way in.** It showed an insert + sitting for exactly `socketTimeoutMS` against an idle engine, which is what + turned a week-long-looking mystery into a five-line fix: + + ```sh + MONGODB_LOG_COMMAND=debug MONGODB_LOG_PATH=stderr \ + node tests/spec/run.js --skip 68 --limit 2 2>drv.log + ``` + +`--skip`/`--limit` exist for exactly this: the collapse reduced to a +reproducible two-file window, which is what made it tractable. + +Still worth recording the baseline on an otherwise idle machine, and do not +tighten `--op-timeout-ms` to make a run finish sooner — a tight budget turns +load into apparent engine failures, which is how I misdiagnosed this once +already. diff --git a/tests/spec/fetch.sh b/tests/spec/fetch.sh new file mode 100755 index 0000000..3d4d554 --- /dev/null +++ b/tests/spec/fetch.sh @@ -0,0 +1,44 @@ +#!/usr/bin/env bash +# Fetch the pinned mongodb/specifications test suites. +# +# Pinned, not vendored: the suites are ~175 JSON files that upstream rewrites +# wholesale, and a pinned commit gives the same reproducibility as vendoring +# without putting someone else's test corpus in this repo's history. PLAN D2 +# requires the pin; `specifications/` is gitignored. +# +# Bump SPEC_COMMIT deliberately, in its own commit, and re-record the +# scorecard in the same commit -- otherwise a scorecard delta cannot be told +# apart from an upstream change. +set -euo pipefail + +SPEC_COMMIT="615e0f9ca5f554614636c098225fbbf1be55565d" +SPEC_REPO="https://github.com/mongodb/specifications.git" + +cd "$(dirname "$0")" +DEST="specifications" + +if [ -d "$DEST/.git" ] && [ "$(git -C "$DEST" rev-parse HEAD 2>/dev/null || true)" = "$SPEC_COMMIT" ]; then + echo "specs already at $SPEC_COMMIT" + exit 0 +fi + +rm -rf "$DEST" +mkdir -p "$DEST" +cd "$DEST" + +# Sparse + depth 1: the full history is ~100 MB and we need two directories. +git init -q . +git remote add origin "$SPEC_REPO" +git config core.sparseCheckout true +mkdir -p .git/info +cat > .git/info/sparse-checkout <<'EOF' +source/crud/tests/ +source/unified-test-format/ +EOF + +echo "fetching $SPEC_COMMIT ..." +git fetch -q --depth 1 origin "$SPEC_COMMIT" +git checkout -q FETCH_HEAD + +n=$(find source/crud/tests/unified -name '*.json' | wc -l | tr -d ' ') +echo "specs at $SPEC_COMMIT ($n crud/unified json files)" diff --git a/tests/spec/run.js b/tests/spec/run.js new file mode 100644 index 0000000..831dbe3 --- /dev/null +++ b/tests/spec/run.js @@ -0,0 +1,743 @@ +'use strict'; +// MongoDB unified-test-format runner, pointed at MultiforaDB. +// +// PLAN D2 makes the official spec suites the gate for command semantics, and +// PLAN D7.6 asks for the harness to exist at M0 with a red baseline -- so this +// is deliberately a *measuring* tool, not a passing one. What it must never do +// is report a pass it did not earn: a compatibility scorecard that flatters +// the engine is worse than no scorecard. Everything unimplemented is reported +// as SKIP with a reason, never as PASS. +// +// bash tests/spec/fetch.sh # pinned suites +// node tests/spec/run.js # spawns its own server +// node tests/spec/run.js --url mongodb://127.0.0.1:27020 # external server +// node tests/spec/run.js --file find.json --verbose +// node tests/spec/run.js --scorecard # rewrite tests/spec/scorecard.txt +// +// Matching follows the pseudo-code in the spec's "Evaluating Matches" +// section verbatim, including the two rules that are easy to get wrong and +// that decide whether a result is a real pass: extra keys are tolerated only +// in a *root* document, and numeric types compare flexibly. + +const fs = require('fs'); +const path = require('path'); +const { spawn } = require('child_process'); + +const DRIVER = path.join(__dirname, '..', 'e2e', 'node_modules', 'mongodb'); +const { MongoClient } = require(DRIVER); +const { EJSON, Long, Int32, Double, Decimal128, ObjectId, Binary, Timestamp } = require(path.join(DRIVER, 'lib', 'bson.js')); + +const SUITE_DIR = path.join(__dirname, 'specifications', 'source', 'crud', 'tests', 'unified'); +const SCORECARD = path.join(__dirname, 'scorecard.txt'); +const REPO = path.join(__dirname, '..', '..'); + +// The runner implements schema 1.0-1.9 features that CRUD tests actually use. +// A file declaring more than this is skipped whole rather than half-run. +const MAX_SCHEMA = [1, 24]; + +const argv = process.argv.slice(2); +function opt(name, dflt) { + const i = argv.indexOf('--' + name); + if (i < 0) return dflt; + const v = argv[i + 1]; + return v === undefined || v.startsWith('--') ? true : v; +} +const VERBOSE = !!opt('verbose', false); +const ONLY_FILE = opt('file', null); +const WRITE_SCORECARD = !!opt('scorecard', false); +const EXTERNAL_URL = opt('url', null); +const PORT = parseInt(opt('port', '27222'), 10); +const BIN = process.env.MFDB_BIN || path.join(REPO, 'zig-out', 'bin', 'multiforadb'); +const DB_PATH = process.env.MFDB_SPEC_DB || path.join(REPO, '.zig-cache', 'spec.log'); + +// --------------------------------------------------------------------------- +// Server lifecycle (mirrors tests/e2e/e2e6.js: poll with a real ping, and +// treat an exited child as a hard failure so we cannot silently measure a +// stale server on the same port). +// --------------------------------------------------------------------------- + +let server = null; +let serverDead = false; +let serverExit = null; +let serverOut = []; +// Persisted, not just buffered in memory: when the server dies mid-run its +// panic is the only thing that explains the hundreds of ECONNREFUSED failures +// that follow, and an in-memory buffer is lost if the runner itself is killed. +const SERVER_LOG = path.join(REPO, '.zig-cache', 'spec-server.out'); + +function sleep(ms) { return new Promise((r) => setTimeout(r, ms)); } + +// A single case must not be able to stall the baseline run. A timeout is +// reported as a FAIL, not a SKIP: "the engine never answered" is a result, and +// hiding it in the skip column would overstate compatibility. +// +// Per-operation, enforced by the driver (CSOT `timeoutMS`) so the promise +// settles instead of being abandoned. The budget exists because some commands +// produce no reply at all rather than an error, and a driver would otherwise +// wait indefinitely. +// +// 10 s, not the 3 s that seems ample for an in-process engine on loopback +// where a clean suite file runs in well under a second. A tight budget makes +// the scorecard a function of machine load rather than of the engine: with a +// concurrent `zig build test` (which compiles and runs multi-threaded tests), +// a 3 s budget turned 166 perfectly good cases into timeout FAILs, and the +// same files passed alone. A generous budget costs wall-clock only on cases +// that genuinely hang, which are few. Don't tighten it to speed up a run. +const OP_TIMEOUT_MS = parseInt(opt('op-timeout-ms', '10000'), 10); +// Backstop only, for a hang the driver's own timeout cannot see. It should +// essentially never fire -- when it does, the case is abandoned, and abandoned +// work is what corrupted two earlier baselines, so it is deliberately far +// above OP_TIMEOUT_MS rather than close to it. +const CASE_TIMEOUT_MS = parseInt(opt('case-timeout-ms', '60000'), 10); + +// Live handle census. Two baselines were wrecked by the runner accumulating +// something across a single process's 175 files, and a count taken only at the +// end could not say *when* it started. Printed per file so the transition is +// visible in the log. +function resourceTag() { + if (!process.getActiveResourcesInfo) return ''; + const counts = {}; + for (const r of process.getActiveResourcesInfo()) counts[r] = (counts[r] || 0) + 1; + const interesting = Object.entries(counts) + .filter(([k]) => k !== 'Immediate' && k !== 'TTYWrap' && k !== 'ProcessWrap' && k !== 'PipeWrap') + .map(([k, v]) => `${k}=${v}`) + .join(' '); + return interesting ? ` [${interesting}]` : ''; +} + +// Server round-trip time, so "the engine is getting slower as the run goes on" +// is a number in the log rather than a guess. Also counts databases, since +// state accumulating on the server is the obvious candidate. +async function pingTag() { + try { + const t0 = process.hrtime.bigint(); + await harness.db('admin').command({ ping: 1 }); + const ms = Number(process.hrtime.bigint() - t0) / 1e6; + let ndbs = '?'; + try { + const r = await harness.db('admin').command({ listDatabases: 1, nameOnly: true }); + ndbs = String((r.databases || []).length); + } catch (e) { + ndbs = `ERR(${e.name}: ${String(e.message).slice(0, 60)})`; + } + return `ping=${ms.toFixed(1)}ms dbs=${ndbs}`; + } catch (e) { + return `ping=FAILED(${e.name})`; + } +} + +async function withTimeout(fn, ms, msg) { + let timer; + const guard = new Promise((_, reject) => { timer = setTimeout(() => reject(new Error(msg)), ms); }); + try { return await Promise.race([fn(), guard]); } finally { clearTimeout(timer); } +} + +async function startServer() { + try { fs.unlinkSync(DB_PATH); } catch (e) { /* first run */ } + if (!fs.existsSync(BIN)) throw new Error(`${BIN} missing - run \`zig build\` first`); + server = spawn(BIN, ['--port', String(PORT), '--db', DB_PATH, '--ttl-sweep-secs', '0'], { stdio: ['ignore', 'pipe', 'pipe'] }); + try { fs.unlinkSync(SERVER_LOG); } catch (e) { /* first run */ } + const sink = fs.createWriteStream(SERVER_LOG, { flags: 'a' }); + const grab = (b) => { sink.write(b); serverOut.push(b.toString()); if (serverOut.length > 400) serverOut.shift(); }; + server.stdout.on('data', grab); + server.stderr.on('data', grab); + server.on('exit', (code, signal) => { serverDead = true; serverExit = signal ? `signal ${signal}` : `exit code ${code}`; }); + + const url = `mongodb://127.0.0.1:${PORT}`; + const deadline = Date.now() + 20000; + while (Date.now() < deadline) { + if (serverDead) throw new Error('server exited during startup:\n' + serverOut.join('')); + try { + const c = new MongoClient(url, { serverSelectionTimeoutMS: 500 }); + await c.connect(); + await c.db('admin').command({ ping: 1 }); + await c.close(); + return url; + } catch (e) { await sleep(100); } + } + throw new Error('server did not become ready:\n' + serverOut.join('')); +} + +function stopServer() { + if (server && !serverDead) server.kill('SIGKILL'); +} +process.on('exit', stopServer); +process.on('SIGINT', () => { stopServer(); process.exit(130); }); + +// --------------------------------------------------------------------------- +// Matching -- the spec's Evaluating Matches algorithm. +// --------------------------------------------------------------------------- + +class MatchError extends Error {} + +function fail(pathStr, msg) { throw new MatchError(`${pathStr || ''}: ${msg}`); } + +function isPlainDoc(v) { + if (v === null || typeof v !== 'object' || Array.isArray(v)) return false; + // BSON scalars are objects too; they are values, not documents. + return !(v instanceof ObjectId || v instanceof Binary || v instanceof Long || + v instanceof Int32 || v instanceof Double || v instanceof Decimal128 || + v instanceof Timestamp || v instanceof Date || v instanceof RegExp); +} + +function specialKey(doc) { + if (!isPlainDoc(doc)) return null; + const ks = Object.keys(doc); + return ks.length === 1 && ks[0].startsWith('$$') ? ks[0] : null; +} + +function numeric(v) { + if (typeof v === 'number') return v; + if (v instanceof Int32 || v instanceof Double) return v.valueOf(); + if (v instanceof Long) return Number(v.toString()); + return null; +} + +// The spec's type aliases, restricted to what the CRUD suites use. +function matchesType(actual, alias) { + switch (alias) { + case 'double': return typeof actual === 'number' || actual instanceof Double; + case 'int': return actual instanceof Int32 || (typeof actual === 'number' && Number.isInteger(actual)); + case 'long': return actual instanceof Long || (typeof actual === 'number' && Number.isInteger(actual)); + case 'string': return typeof actual === 'string'; + case 'object': return isPlainDoc(actual); + case 'array': return Array.isArray(actual); + case 'binData': return actual instanceof Binary; + case 'objectId': return actual instanceof ObjectId; + case 'bool': return typeof actual === 'boolean'; + case 'date': return actual instanceof Date; + case 'null': return actual === null; + case 'regex': return actual instanceof RegExp; + case 'timestamp': return actual instanceof Timestamp; + case 'decimal': return actual instanceof Decimal128; + case 'number': return numeric(actual) !== null || actual instanceof Decimal128; + default: return null; // unknown alias -> caller reports unsupported + } +} + +function scalarEqual(expected, actual) { + const en = numeric(expected), an = numeric(actual); + if (en !== null && an !== null) return en === an; // flexible numerics + if (expected === null) return actual === null; + if (expected instanceof Date) return actual instanceof Date && +expected === +actual; + if (expected instanceof ObjectId) return actual instanceof ObjectId && expected.equals(actual); + if (expected instanceof Binary) return actual instanceof Binary && Buffer.compare(expected.buffer, actual.buffer) === 0 && expected.sub_type === actual.sub_type; + if (expected instanceof Decimal128) return actual instanceof Decimal128 && expected.toString() === actual.toString(); + if (expected instanceof Timestamp) return actual instanceof Timestamp && expected.equals(actual); + if (expected instanceof RegExp) return actual instanceof RegExp && String(expected) === String(actual); + if (typeof expected !== typeof actual) return false; + return expected === actual; +} + +// `root` implements the one rule that separates a real pass from a lenient +// one: only a root document may carry keys the expectation does not mention. +function match(expected, actual, entities, pathStr = '', root = true) { + const sk = specialKey(expected); + if (sk) return special(sk, expected[sk], actual, entities, pathStr, true); + + if (isPlainDoc(expected)) { + if (!isPlainDoc(actual)) fail(pathStr, `expected a document, got ${describe(actual)}`); + for (const [k, v] of Object.entries(expected)) { + const kp = pathStr ? `${pathStr}.${k}` : k; + const vsk = specialKey(v); + if (vsk) { + const consumed = special(vsk, v[vsk], actual[k], entities, kp, Object.prototype.hasOwnProperty.call(actual, k)); + if (!consumed) continue; + continue; + } + if (!Object.prototype.hasOwnProperty.call(actual, k)) fail(kp, 'missing from actual'); + match(v, actual[k], entities, kp, false); + } + if (!root) { + const extra = Object.keys(actual).filter((k) => !Object.prototype.hasOwnProperty.call(expected, k)); + if (extra.length) fail(pathStr, `unexpected extra keys ${JSON.stringify(extra)}`); + } + return; + } + + if (Array.isArray(expected)) { + if (!Array.isArray(actual)) fail(pathStr, `expected an array, got ${describe(actual)}`); + if (expected.length !== actual.length) fail(pathStr, `expected ${expected.length} elements, got ${actual.length}`); + // Array elements are not root documents: `expectResult: [ {...} ]` from + // find must match its documents exactly, extra fields included. + expected.forEach((e, i) => match(e, actual[i], entities, `${pathStr}[${i}]`, false)); + return; + } + + if (!scalarEqual(expected, actual)) fail(pathStr, `expected ${describe(expected)}, got ${describe(actual)}`); +} + +// Returns whether the actual value still has to be matched by the caller. +function special(op, arg, actual, entities, pathStr, present) { + switch (op) { + case '$$exists': + if (arg && !present) fail(pathStr, 'expected the key to exist'); + if (!arg && present) fail(pathStr, 'expected the key to be absent'); + return false; + case '$$type': { + const aliases = Array.isArray(arg) ? arg : [arg]; + const results = aliases.map((a) => matchesType(actual, a)); + if (results.some((r) => r === null)) throw new Unsupported(`$$type alias ${JSON.stringify(arg)}`); + if (!results.some((r) => r === true)) fail(pathStr, `expected type ${JSON.stringify(arg)}, got ${describe(actual)}`); + return false; + } + case '$$unsetOrMatches': + if (!present || actual === undefined) return false; + match(arg, actual, entities, pathStr, false); + return false; + case '$$matchesEntity': { + if (!(arg in entities.map)) fail(pathStr, `entity ${arg} not found`); + match(entities.map[arg], actual, entities, pathStr, false); + return false; + } + case '$$matchesHexBytes': + if (!(actual instanceof Binary)) fail(pathStr, 'expected binary data'); + if (actual.buffer.toString('hex') !== String(arg).toLowerCase()) fail(pathStr, 'hex bytes differ'); + return false; + default: + throw new Unsupported(`match operator ${op}`); + } +} + +function describe(v) { + if (v === undefined) return 'undefined'; + if (v === null) return 'null'; + try { return EJSON.stringify(v, { relaxed: true }); } catch (e) { return String(v); } +} + +class Unsupported extends Error {} + +// --------------------------------------------------------------------------- +// Operations +// --------------------------------------------------------------------------- + +// Argument keys the spec passes positionally rather than as driver options. +const POSITIONAL = new Set(['filter', 'document', 'documents', 'update', 'replacement', 'pipeline', 'fieldName', 'models', 'requests', 'keys', 'name', 'indexes', 'command', 'session', 'entity', 'to']); + +function options(args, drop = []) { + const o = {}; + for (const [k, v] of Object.entries(args || {})) { + if (POSITIONAL.has(k) || drop.includes(k)) continue; + o[k] = v; + } + return Object.keys(o).length ? o : undefined; +} + +function requireNoSession(args) { + if (args && args.session) throw new Unsupported('explicit sessions (M4)'); +} + +async function runOperation(op, entities) { + const args = op.arguments || {}; + requireNoSession(args); + const target = entities.map[op.object]; + if (op.object === 'testRunner') throw new Unsupported(`testRunner operation ${op.name}`); + if (target === undefined) throw new Unsupported(`entity ${op.object}`); + + switch (op.name) { + // -- read ---------------------------------------------------------- + case 'find': return await target.find(args.filter || {}, options(args)).toArray(); + case 'findOne': return await target.findOne(args.filter || {}, options(args)); + case 'aggregate': return await target.aggregate(args.pipeline, options(args)).toArray(); + case 'countDocuments': return await target.countDocuments(args.filter || {}, options(args)); + case 'estimatedDocumentCount': return await target.estimatedDocumentCount(options(args)); + case 'distinct': return await target.distinct(args.fieldName, args.filter || {}, options(args)); + case 'listIndexes': return await target.listIndexes(options(args)).toArray(); + case 'listIndexNames': return await target.indexes(options(args)).then((ix) => ix.map((i) => i.name)); + case 'listCollections': return await target.listCollections(args.filter || {}, options(args)).toArray(); + case 'listCollectionNames': return await target.listCollections(args.filter || {}, options(args)).toArray().then((cs) => cs.map((c) => c.name)); + case 'runCommand': return await target.command(args.command, options(args, ['commandName'])); + + // -- write --------------------------------------------------------- + case 'insertOne': return plain(await target.insertOne(args.document, options(args))); + case 'insertMany': return plain(await target.insertMany(args.documents, options(args))); + case 'updateOne': return plain(await target.updateOne(args.filter, args.update, options(args))); + case 'updateMany': return plain(await target.updateMany(args.filter, args.update, options(args))); + case 'replaceOne': return plain(await target.replaceOne(args.filter, args.replacement, options(args))); + case 'deleteOne': return plain(await target.deleteOne(args.filter, options(args))); + case 'deleteMany': return plain(await target.deleteMany(args.filter, options(args))); + case 'bulkWrite': return plain(await target.bulkWrite(args.requests, options(args))); + case 'findOneAndUpdate': return unwrapFam(await target.findOneAndUpdate(args.filter, args.update, options(args))); + case 'findOneAndReplace': return unwrapFam(await target.findOneAndReplace(args.filter, args.replacement, options(args))); + case 'findOneAndDelete': return unwrapFam(await target.findOneAndDelete(args.filter, options(args))); + + // -- collection / index management ---------------------------------- + case 'createIndex': return await target.createIndex(args.keys, options(args)); + case 'dropIndex': return await target.dropIndex(args.name, options(args)); + case 'createCollection': return void (await target.createCollection(args.collection, options(args, ['collection']))); + case 'dropCollection': return void (await target.dropCollection(args.collection, options(args, ['collection']))); + case 'assertCollectionExists': case 'assertCollectionNotExists': + throw new Unsupported(`testRunner operation ${op.name}`); + default: + throw new Unsupported(`operation ${op.name}`); + } +} + +// Driver result objects carry class instances and `acknowledged`; the spec +// expects plain documents. Turning them into plain objects keeps the +// root-document rule meaningful (extra keys tolerated only at the root). +function plain(r) { + if (r === null || typeof r !== 'object') return r; + const o = {}; + for (const k of Object.keys(r)) { + if (k === 'acknowledged') continue; + o[k] = r[k]; + } + if (typeof r.getUpsertedIds === 'function') { + // BulkWriteResult exposes counts through getters, not own keys. + for (const k of ['insertedCount', 'matchedCount', 'modifiedCount', 'deletedCount', 'upsertedCount', 'insertedIds', 'upsertedIds']) { + if (r[k] !== undefined) o[k] = r[k]; + } + } + return o; +} + +// Driver 5+ returns the document itself; 4.x wrapped it in `{value}`. e2e.js +// tolerates both the same way (`fam.value ?? fam`). +function unwrapFam(r) { + if (r && typeof r === 'object' && 'value' in r && 'ok' in r) return r.value; + return r; +} + +// --------------------------------------------------------------------------- +// runOnRequirements +// --------------------------------------------------------------------------- + +function cmpVersion(a, b) { + const pa = String(a).split('.').map(Number), pb = String(b).split('.').map(Number); + for (let i = 0; i < Math.max(pa.length, pb.length); i++) { + const x = pa[i] || 0, y = pb[i] || 0; + if (x !== y) return x < y ? -1 : 1; + } + return 0; +} + +function unmetRequirement(reqs, server) { + if (!reqs) return null; + const reasons = []; + for (const r of reqs) { + const why = []; + if (r.minServerVersion && cmpVersion(server.version, r.minServerVersion) < 0) why.push(`needs server >= ${r.minServerVersion}`); + if (r.maxServerVersion && cmpVersion(server.version, r.maxServerVersion) > 0) why.push(`needs server <= ${r.maxServerVersion}`); + if (r.topologies && !r.topologies.includes('single')) why.push(`needs topology ${r.topologies.join('/')}`); + if (r.auth === true) why.push('needs auth (M7)'); + if (r.csfle) why.push('needs csfle'); + if (r.serverless === 'require') why.push('needs serverless'); + if (r.serverParameters) why.push('needs server parameters'); + if (why.length === 0) return null; // this alternative is satisfied + reasons.push(why.join(', ')); + } + return reasons[0] || 'unmet requirement'; +} + +// --------------------------------------------------------------------------- +// One file +// --------------------------------------------------------------------------- + +// One client for the whole run, used for seeding and outcome checks. +// +// It used to be one client per seed and per outcome check. That, plus per-test +// entity clients that leaked whenever a case timed out before buildEntities +// returned, meant hundreds of live MongoClients accumulated -- each with its +// own pool and heartbeat timer. Node's event loop starved, the per-case timer +// then fired before operations could complete, and every case from that point +// on "failed" with a timeout. A ping from a separate process still answered, +// which made it look convincingly like a server-side wedge. It was the runner. +// +// The lesson worth keeping: a harness whose own failures look like engine +// failures will silently invent a compatibility gap. Hence also the assertion +// below that clients never accumulate. +let harness = null; + +async function seedInitialData(initialData) { + if (!initialData) return; + for (const spec of initialData) { + const coll = harness.db(spec.databaseName).collection(spec.collectionName); + await coll.drop().catch(() => { }); + if (spec.documents && spec.documents.length) await coll.insertMany(spec.documents); + else await harness.db(spec.databaseName).createCollection(spec.collectionName).catch(() => { }); + } +} + +// `clients` is supplied by the caller so that entities created before a +// failure are still closed: returning them only on success is what leaked. +async function buildEntities(url, createEntities, clients) { + const map = {}; + for (const spec of createEntities || []) { + const [kind, def] = Object.entries(spec)[0]; + switch (kind) { + case 'client': { + if (def.useMultipleMongoses === true) { /* single topology: harmless */ } + // `timeoutMS` (CSOT) makes each operation reject on its own + // rather than being abandoned by an outer race. That matters + // more than it looks: an abandoned operation keeps running, + // keeps its client alive, and -- since it may still be inside + // buildEntities -- can create further clients *after* cleanup + // has already run. That is what leaked, and what turned into + // 190 phantom timeout FAILs. + const c = new MongoClient(url, Object.assign({ + serverSelectionTimeoutMS: 2000, + connectTimeoutMS: 2000, + timeoutMS: OP_TIMEOUT_MS, + }, def.uriOptions || {})); + // Registered before connect, so a client whose connect throws + // or is abandoned is still closed by the caller. + clients.push(c); + await c.connect(); + if (clients.abandoned) { + // Cleanup already ran; this client would otherwise linger. + await c.close().catch(() => { }); + throw new Error('case abandoned'); + } + map[def.id] = c; + break; + } + case 'database': map[def.id] = map[def.client].db(def.databaseName); break; + case 'collection': map[def.id] = map[def.database].collection(def.collectionName); break; + case 'session': throw new Unsupported('session entities (M4)'); + case 'bucket': throw new Unsupported('gridfs bucket entities'); + case 'clientEncryption': throw new Unsupported('client-side encryption'); + default: throw new Unsupported(`entity type ${kind}`); + } + } + return { map, clients }; +} + +async function verifyOutcome(outcome, entities) { + if (!outcome) return; + for (const spec of outcome) { + const actual = await harness.db(spec.databaseName).collection(spec.collectionName) + .find({}, { sort: { _id: 1 } }).toArray(); + match(spec.documents, actual, entities, `outcome ${spec.databaseName}.${spec.collectionName}`, false); + } +} + +async function runFile(file, url, server) { + const text = fs.readFileSync(path.join(SUITE_DIR, file), 'utf8'); + const doc = EJSON.parse(text, { relaxed: false }); + const out = { file, pass: 0, fail: 0, skip: 0, cases: [] }; + + const schema = String(doc.schemaVersion || '1.0').split('.').map(Number); + if (schema[0] > MAX_SCHEMA[0] || (schema[0] === MAX_SCHEMA[0] && (schema[1] || 0) > MAX_SCHEMA[1])) { + out.skip = (doc.tests || []).length; + out.cases.push({ name: '*', status: 'SKIP', reason: `schemaVersion ${doc.schemaVersion} > ${MAX_SCHEMA.join('.')}` }); + return out; + } + const fileUnmet = unmetRequirement(doc.runOnRequirements, server); + if (fileUnmet) { + out.skip = (doc.tests || []).length; + out.cases.push({ name: '*', status: 'SKIP', reason: fileUnmet }); + return out; + } + + for (const test of doc.tests || []) { + const name = test.description; + if (test.skipReason) { out.skip++; out.cases.push({ name, status: 'SKIP', reason: 'upstream skipReason: ' + test.skipReason }); continue; } + const unmet = unmetRequirement(test.runOnRequirements, server); + if (unmet) { out.skip++; out.cases.push({ name, status: 'SKIP', reason: unmet }); continue; } + + // Owned out here, not by buildEntities, so a case that dies partway + // still has every client it managed to open closed below. + const clients = []; + try { + await withTimeout(async () => { + await seedInitialData(doc.initialData); + const entities = await buildEntities(url, doc.createEntities, clients); + for (const op of test.operations) await runOne(op, entities); + await verifyOutcome(test.outcome, entities); + }, CASE_TIMEOUT_MS, `case exceeded ${CASE_TIMEOUT_MS} ms`); + out.pass++; + out.cases.push({ name, status: 'PASS' }); + } catch (e) { + if (e instanceof Unsupported) { out.skip++; out.cases.push({ name, status: 'SKIP', reason: 'runner: ' + e.message }); } + else { out.fail++; out.cases.push({ name, status: 'FAIL', reason: (e instanceof MatchError ? '' : (e.constructor.name + ': ')) + e.message.split('\n')[0].slice(0, 220) }); } + } finally { + // Unconditional, and the flag matters: if the case was abandoned, + // its continuation may still be running and about to open another + // client, which buildEntities closes itself on seeing this. + clients.abandoned = true; + for (const c of clients) await c.close().catch(() => { }); + } + } + return out; +} + +async function runOne(op, entities) { + if (op.name === 'failPoint' || op.name === 'targetedFailPoint') throw new Unsupported('failPoint'); + let result, err = null; + try { + result = await runOperation(op, entities); + } catch (e) { + if (e instanceof Unsupported) throw e; + err = e; + } + + if (op.expectError) { + if (!err) fail(op.name, 'expected an error, the operation succeeded'); + checkError(op.expectError, err, entities, op.name); + return; + } + if (err) throw err; + if (op.ignoreResultAndError) return; + if ('expectResult' in op) match(op.expectResult, result, entities, op.name, true); + if (op.saveResultAsEntity) entities.map[op.saveResultAsEntity] = result; +} + +function checkError(exp, err, entities, where) { + if (exp.isError === false) fail(where, 'expected no error'); + if (exp.isClientError === true && err.code !== undefined && err.code !== null) { + fail(where, `expected a client-side error, got server code ${err.code}`); + } + if (exp.errorContains) { + const hay = String(err.message || '').toLowerCase(); + if (!hay.includes(String(exp.errorContains).toLowerCase())) fail(where, `error message ${JSON.stringify(String(err.message).slice(0, 120))} does not contain ${JSON.stringify(exp.errorContains)}`); + } + if (exp.errorCode !== undefined) { + const actual = numeric(err.code); + if (actual !== numeric(exp.errorCode)) fail(where, `expected error code ${exp.errorCode}, got ${err.code}`); + } + if (exp.errorCodeName !== undefined) { + const actual = err.codeName || (err.result && err.result.codeName); + if (actual !== exp.errorCodeName) fail(where, `expected codeName ${exp.errorCodeName}, got ${actual}`); + } + if (exp.errorLabelsContain) { + for (const l of exp.errorLabelsContain) if (!(err.errorLabels || []).includes(l)) fail(where, `expected error label ${l}`); + } + if (exp.errorLabelsOmit) { + for (const l of exp.errorLabelsOmit) if ((err.errorLabels || []).includes(l)) fail(where, `expected no error label ${l}`); + } + if (exp.errorResponse) { + const resp = err.result || err.errorResponse || err; + match(exp.errorResponse, resp, entities, where + '.errorResponse', true); + } + if (exp.writeErrors || exp.writeConcernErrors) throw new Unsupported('clientBulkWrite error assertions'); + if ('expectResult' in exp) { + const r = err.result || (err.writeErrors ? plain(err) : undefined); + if (r === undefined) throw new Unsupported('expectResult on an error the driver does not expose'); + match(exp.expectResult, r, entities, where + '.expectResult', true); + } +} + +// --------------------------------------------------------------------------- +// main +// --------------------------------------------------------------------------- + +(async () => { + if (!fs.existsSync(SUITE_DIR)) { + console.error(`missing ${SUITE_DIR}\nrun: bash tests/spec/fetch.sh`); + process.exit(2); + } + let files = fs.readdirSync(SUITE_DIR).filter((f) => f.endsWith('.json')) + .filter((f) => !ONLY_FILE || f === ONLY_FILE || f === ONLY_FILE + '.json') + .sort(); + // --skip/--limit exist for bisecting a run whose failures depend on + // position rather than content, which has happened more than once. + const skip = parseInt(opt('skip', '0'), 10); + if (skip > 0) files = files.slice(skip); + const limit = parseInt(opt('limit', '0'), 10); + if (limit > 0) files = files.slice(0, limit); + if (!files.length) { console.error('no matching suite files'); process.exit(2); } + + const url = EXTERNAL_URL || await startServer(); + + harness = new MongoClient(url, { serverSelectionTimeoutMS: 3000, connectTimeoutMS: 3000, socketTimeoutMS: 8000 }); + await harness.connect(); + const bi = await harness.db('admin').command({ buildInfo: 1 }).catch(() => ({ version: '0.0.0' })); + const hello = await harness.db('admin').command({ hello: 1 }).catch(() => ({})); + const server = { version: bi.version || '0.0.0', maxWireVersion: hello.maxWireVersion }; + + const results = []; + let died = null; + for (const f of files) { + // A dead server turns every remaining case into ECONNREFUSED, which + // would land in the scorecard as ~700 engine failures and bury the one + // fact that matters: it crashed, and where. Stop at the crash instead. + if (serverDead && !EXTERNAL_URL) { + died = { after: results.length ? results[results.length - 1].file : '', next: f }; + break; + } + let r; + try { + r = await runFile(f, url, server); + } catch (e) { + r = { file: f, pass: 0, fail: 0, skip: 0, cases: [{ name: '*', status: 'ERROR', reason: e.message.split('\n')[0].slice(0, 200) }], errored: true }; + } + results.push(r); + const tag = r.errored ? 'ERROR' : `${r.pass} pass, ${r.fail} fail, ${r.skip} skip`; + console.log(`${f.padEnd(48)} ${tag}${resourceTag()} ${await pingTag()}`); + if (VERBOSE) for (const c of r.cases) if (c.status !== 'PASS') console.log(` ${c.status.padEnd(5)} ${c.name}${c.reason ? ' -- ' + c.reason : ''}`); + } + + const tot = results.reduce((a, r) => ({ pass: a.pass + r.pass, fail: a.fail + r.fail, skip: a.skip + r.skip }), { pass: 0, fail: 0, skip: 0 }); + const errored = results.filter((r) => r.errored).length; + console.log(`\ncrud+aggregate: ${tot.pass} pass, ${tot.fail} fail, ${tot.skip} skip across ${results.length}/${files.length} files (${errored} files errored)`); + + if (died) { + console.error(`\n*** THE SERVER DIED (${serverExit}) ***`); + console.error(`last file completed: ${died.after}; would have run next: ${died.next}`); + console.error(`server output: ${path.relative(REPO, SERVER_LOG)}`); + console.error('--- tail ---\n' + serverOut.slice(-40).join('')); + console.error('Refusing to write a scorecard from a partial run.'); + process.exit(3); + } + + if (WRITE_SCORECARD) { + fs.writeFileSync(SCORECARD, scorecardText(results, tot, errored, server, files.length)); + console.log(`wrote ${path.relative(REPO, SCORECARD)}`); + } + // A runner leak once turned into ~77 phantom "engine timeouts" (see the + // comment on `harness`). This is the tripwire for that class of bug: if the + // harness is the only client left standing, nothing accumulated. + const leaked = process.getActiveResourcesInfo + ? process.getActiveResourcesInfo().filter((r) => r === 'Timeout').length + : 0; + if (leaked > 8) { + console.error(`\nWARNING: ${leaked} timers still active at the end of the run.`); + console.error('That is how a client leak looks; treat the timeout FAILs above as suspect.'); + } + await harness.close(true).catch(() => { }); + + // The runner's own exit code reports whether it *ran*, not whether the + // engine passed -- M0's gate is a recorded baseline, and a red baseline is + // the expected state. A non-zero exit here would make it useless as a gate. + console.log('SPEC_RUNNER_OK'); + stopServer(); + process.exit(0); +})().catch((e) => { + console.error('spec runner failed:', e); + if (serverOut.length) console.error('--- server output ---\n' + serverOut.slice(-30).join('')); + stopServer(); + process.exit(1); +}); + +function scorecardText(results, tot, errored, server, nfiles) { + const L = []; + L.push('# MongoDB spec-test scorecard (PLAN D2) -- crud + aggregate, unified format'); + L.push(`# specs: mongodb/specifications @ 615e0f9 (pinned in tests/spec/fetch.sh)`); + L.push(`# driver: mongodb@${require(path.join(DRIVER, 'package.json')).version} (pinned in tests/e2e/package-lock.json)`); + L.push(`# server: MultiforaDB reporting version ${server.version}, maxWireVersion ${server.maxWireVersion}`); + L.push('# reproduce: bash tests/spec/fetch.sh && node tests/spec/run.js --scorecard'); + L.push('#'); + L.push('# What SKIP means here, so the totals are not read as better than they are:'); + L.push('# - the suite needs a feature whose milestone has not landed (sessions M4,'); + L.push('# auth M7, failPoints, gridfs) -- counted as skip, never as pass;'); + L.push('# - or the runner itself does not implement the operation/matcher yet.'); + L.push('# Deliberately NOT asserted yet: expectEvents (command monitoring). Those'); + L.push('# assertions are about driver-visible command shape rather than result'); + L.push('# semantics; ignoring them makes some cases pass that a full runner would'); + L.push('# fail, so treat `pass` as an upper bound until M1 wires events up.'); + L.push(''); + L.push(`total\t${tot.pass} pass\t${tot.fail} fail\t${tot.skip} skip\t${nfiles} files\t${errored} errored`); + L.push(''); + L.push('# per-file: name\tpass\tfail\tskip'); + for (const r of results) L.push(`${r.file}\t${r.pass}\t${r.fail}\t${r.skip}${r.errored ? '\terrored' : ''}`); + L.push(''); + L.push('# every non-passing case, with its reason'); + for (const r of results) { + for (const c of r.cases) { + if (c.status === 'PASS') continue; + L.push(`${r.file}\t${c.status}\t${c.name}\t${c.reason || ''}`); + } + } + return L.join('\n') + '\n'; +} diff --git a/tests/spec/scorecard.txt b/tests/spec/scorecard.txt new file mode 100644 index 0000000..c5dd7a4 --- /dev/null +++ b/tests/spec/scorecard.txt @@ -0,0 +1,503 @@ +# MongoDB spec-test scorecard (PLAN D2) -- crud + aggregate, unified format +# specs: mongodb/specifications @ 615e0f9 (pinned in tests/spec/fetch.sh) +# driver: mongodb@7.5.0 (pinned in tests/e2e/package-lock.json) +# server: MultiforaDB reporting version 4.4.0, maxWireVersion 8 +# reproduce: bash tests/spec/fetch.sh && node tests/spec/run.js --scorecard +# +# What SKIP means here, so the totals are not read as better than they are: +# - the suite needs a feature whose milestone has not landed (sessions M4, +# auth M7, failPoints, gridfs) -- counted as skip, never as pass; +# - or the runner itself does not implement the operation/matcher yet. +# Deliberately NOT asserted yet: expectEvents (command monitoring). Those +# assertions are about driver-visible command shape rather than result +# semantics; ignoring them makes some cases pass that a full runner would +# fail, so treat `pass` as an upper bound until M1 wires events up. + +total 131 pass 161 fail 195 skip 175 files 0 errored + +# per-file: name pass fail skip +aggregate-allowdiskuse.json 3 0 0 +aggregate-collation.json 0 1 0 +aggregate-let.json 0 2 2 +aggregate-merge-errorResponse.json 0 0 1 +aggregate-merge.json 0 5 0 +aggregate-out-readConcern.json 0 0 4 +aggregate-out.json 0 2 0 +aggregate-rawdata.json 1 0 1 +aggregate-write-readPreference.json 0 0 4 +aggregate.json 5 0 2 +bulkWrite-arrayFilters.json 0 3 0 +bulkWrite-collation.json 0 2 0 +bulkWrite-comment.json 0 2 1 +bulkWrite-delete-hint-serverError.json 0 0 2 +bulkWrite-delete-hint.json 2 0 0 +bulkWrite-deleteMany-hint-unacknowledged.json 0 2 2 +bulkWrite-deleteMany-let.json 0 1 1 +bulkWrite-deleteMany-rawdata.json 1 0 1 +bulkWrite-deleteOne-hint-unacknowledged.json 0 2 2 +bulkWrite-deleteOne-let.json 0 1 1 +bulkWrite-deleteOne-rawdata.json 1 0 1 +bulkWrite-errorResponse.json 0 0 1 +bulkWrite-insertOne-dots_and_dollars.json 3 1 1 +bulkWrite-replaceOne-dots_and_dollars.json 1 2 1 +bulkWrite-replaceOne-hint-unacknowledged.json 0 2 0 +bulkWrite-replaceOne-let.json 0 1 1 +bulkWrite-replaceOne-rawdata.json 1 0 1 +bulkWrite-replaceOne-sort.json 1 0 1 +bulkWrite-update-hint.json 2 1 0 +bulkWrite-update-validation.json 3 0 0 +bulkWrite-updateMany-dots_and_dollars.json 0 0 4 +bulkWrite-updateMany-hint-unacknowledged.json 0 2 0 +bulkWrite-updateMany-let.json 0 1 1 +bulkWrite-updateMany-pipeline.json 0 1 0 +bulkWrite-updateMany-rawdata.json 0 1 1 +bulkWrite-updateOne-dots_and_dollars.json 0 0 4 +bulkWrite-updateOne-hint-unacknowledged.json 0 2 0 +bulkWrite-updateOne-let.json 0 1 1 +bulkWrite-updateOne-pipeline.json 0 1 0 +bulkWrite-updateOne-rawdata.json 0 1 1 +bulkWrite-updateOne-sort.json 1 0 1 +bulkWrite.json 5 5 0 +bypassDocumentValidation.json 6 3 0 +client-bulkWrite-delete-options.json 0 0 2 +client-bulkWrite-delete-rawdata.json 0 0 2 +client-bulkWrite-errorResponse.json 0 0 1 +client-bulkWrite-errors.json 0 0 9 +client-bulkWrite-mixed-namespaces.json 0 0 1 +client-bulkWrite-options.json 0 0 7 +client-bulkWrite-ordered.json 0 0 3 +client-bulkWrite-partialResults.json 0 0 9 +client-bulkWrite-replaceOne-rawdata.json 0 0 2 +client-bulkWrite-replaceOne-sort.json 0 0 1 +client-bulkWrite-results.json 0 0 3 +client-bulkWrite-update-options.json 0 0 4 +client-bulkWrite-update-pipeline.json 0 0 2 +client-bulkWrite-update-rawdata.json 0 0 2 +client-bulkWrite-update-validation.json 0 0 3 +client-bulkWrite-updateOne-sort.json 0 0 1 +count-collation.json 1 0 1 +count-empty.json 2 0 1 +count-rawdata.json 0 0 2 +count.json 3 1 3 +countDocuments-comment.json 2 0 1 +countDocuments-rawdata.json 1 0 1 +create-null-ids.json 0 6 1 +db-aggregate-rawdata.json 0 1 1 +db-aggregate-write-readPreference.json 0 0 4 +db-aggregate.json 0 2 0 +deleteMany-collation.json 0 1 0 +deleteMany-comment.json 2 0 1 +deleteMany-hint-serverError.json 0 0 2 +deleteMany-hint-unacknowledged.json 0 2 2 +deleteMany-hint.json 2 0 0 +deleteMany-let.json 0 1 1 +deleteMany-rawdata.json 1 0 1 +deleteMany.json 2 0 0 +deleteOne-collation.json 0 1 0 +deleteOne-comment.json 2 0 1 +deleteOne-errorResponse.json 0 0 1 +deleteOne-hint-serverError.json 0 0 2 +deleteOne-hint-unacknowledged.json 0 2 2 +deleteOne-hint.json 2 0 0 +deleteOne-let.json 0 1 1 +deleteOne-rawdata.json 1 0 1 +deleteOne.json 3 0 0 +distinct-collation.json 0 1 0 +distinct-comment.json 1 1 1 +distinct-hint.json 0 0 2 +distinct-rawdata.json 0 1 1 +distinct.json 0 2 0 +estimatedDocumentCount-comment.json 1 1 1 +estimatedDocumentCount-rawdata.json 1 0 1 +estimatedDocumentCount.json 3 1 2 +find-allowdiskuse-serverError.json 0 0 2 +find-allowdiskuse.json 3 0 0 +find-collation.json 0 1 0 +find-comment.json 1 2 2 +find-let.json 0 1 1 +find-rawdata.json 1 0 1 +find.json 2 3 0 +findOne.json 1 1 0 +findOneAndDelete-collation.json 0 1 0 +findOneAndDelete-comment.json 2 0 1 +findOneAndDelete-hint-serverError.json 0 0 2 +findOneAndDelete-hint-unacknowledged.json 0 2 2 +findOneAndDelete-hint.json 2 0 0 +findOneAndDelete-let.json 0 1 1 +findOneAndDelete-rawdata.json 1 0 1 +findOneAndDelete.json 3 0 0 +findOneAndReplace-collation.json 0 1 0 +findOneAndReplace-comment.json 0 2 1 +findOneAndReplace-dots_and_dollars.json 1 2 1 +findOneAndReplace-hint-serverError.json 0 0 2 +findOneAndReplace-hint-unacknowledged.json 0 2 2 +findOneAndReplace-hint.json 0 2 0 +findOneAndReplace-let.json 0 1 1 +findOneAndReplace-rawdata.json 0 1 1 +findOneAndReplace-upsert.json 0 4 0 +findOneAndReplace.json 2 4 0 +findOneAndUpdate-arrayFilters.json 0 3 0 +findOneAndUpdate-collation.json 0 1 0 +findOneAndUpdate-comment.json 0 2 1 +findOneAndUpdate-dots_and_dollars.json 0 0 4 +findOneAndUpdate-errorResponse.json 0 1 1 +findOneAndUpdate-hint-serverError.json 0 0 2 +findOneAndUpdate-hint-unacknowledged.json 0 2 2 +findOneAndUpdate-hint.json 2 0 0 +findOneAndUpdate-let.json 0 1 1 +findOneAndUpdate-pipeline.json 0 1 0 +findOneAndUpdate-rawdata.json 0 1 1 +findOneAndUpdate.json 5 3 0 +insertMany-comment.json 2 0 1 +insertMany-dots_and_dollars.json 0 4 1 +insertMany-rawdata.json 1 0 1 +insertMany.json 2 1 0 +insertOne-comment.json 2 0 1 +insertOne-dots_and_dollars.json 5 3 1 +insertOne-errorResponse.json 0 0 1 +insertOne-rawdata.json 1 0 1 +insertOne.json 1 0 0 +replaceOne-collation.json 0 1 0 +replaceOne-comment.json 0 2 1 +replaceOne-dots_and_dollars.json 2 2 1 +replaceOne-hint-unacknowledged.json 0 2 0 +replaceOne-hint.json 0 2 0 +replaceOne-let.json 0 1 1 +replaceOne-rawdata.json 0 1 1 +replaceOne-sort.json 1 0 1 +replaceOne-validation.json 1 0 0 +replaceOne.json 1 4 0 +updateMany-arrayFilters.json 0 3 0 +updateMany-collation.json 0 1 0 +updateMany-comment.json 2 0 1 +updateMany-dots_and_dollars.json 0 0 4 +updateMany-hint-unacknowledged.json 0 2 0 +updateMany-hint.json 2 0 0 +updateMany-let.json 0 1 1 +updateMany-pipeline.json 0 1 0 +updateMany-rawdata.json 1 0 1 +updateMany-validation.json 1 0 0 +updateMany.json 4 0 0 +updateOne-arrayFilters.json 0 5 0 +updateOne-collation.json 0 1 0 +updateOne-comment.json 2 0 1 +updateOne-dots_and_dollars.json 0 0 4 +updateOne-errorResponse.json 0 0 1 +updateOne-hint-unacknowledged.json 0 2 0 +updateOne-hint.json 2 0 0 +updateOne-let.json 0 1 1 +updateOne-pipeline.json 0 1 0 +updateOne-rawdata.json 1 0 1 +updateOne-sort.json 0 1 1 +updateOne-validation.json 1 0 0 +updateOne.json 4 0 0 + +# every non-passing case, with its reason +aggregate-collation.json FAIL Aggregate with collation aggregate: expected 1 elements, got 0 +aggregate-let.json SKIP Aggregate with let option needs server >= 5.0 +aggregate-let.json FAIL Aggregate with let option unsupported (server-side error) aggregate: expected an error, the operation succeeded +aggregate-let.json SKIP Aggregate to collection with let option needs server >= 5.0 +aggregate-let.json FAIL Aggregate to collection with let option unsupported (server-side error) aggregate: error message "Unrecognized pipeline stage name: '$out'" does not contain "unrecognized field 'let'" +aggregate-merge-errorResponse.json SKIP aggregate $merge DuplicateKey error is accessible needs server >= 5.1 +aggregate-merge.json FAIL Aggregate with $merge MongoServerError: Unrecognized pipeline stage name: '$merge' +aggregate-merge.json FAIL Aggregate with $merge and batch size of 0 MongoServerError: Unrecognized pipeline stage name: '$merge' +aggregate-merge.json FAIL Aggregate with $merge and majority readConcern MongoServerError: Unrecognized pipeline stage name: '$merge' +aggregate-merge.json FAIL Aggregate with $merge and local readConcern MongoServerError: Unrecognized pipeline stage name: '$merge' +aggregate-merge.json FAIL Aggregate with $merge and available readConcern MongoServerError: Unrecognized pipeline stage name: '$merge' +aggregate-out-readConcern.json SKIP * needs topology replicaset/sharded +aggregate-out.json FAIL Aggregate with $out MongoServerError: Unrecognized pipeline stage name: '$out' +aggregate-out.json FAIL Aggregate with $out and batch size of 0 MongoServerError: Unrecognized pipeline stage name: '$out' +aggregate-rawdata.json SKIP Aggregate with rawData option needs server >= 8.2.0 +aggregate-write-readPreference.json SKIP * needs topology replicaset/sharded/load-balanced +aggregate.json SKIP aggregate with a document comment - pre 4.4 needs server <= 4.2.99 +aggregate.json SKIP aggregate with comment does not set comment on getMore - pre 4.4 needs server <= 4.3.99 +bulkWrite-arrayFilters.json FAIL BulkWrite updateOne with arrayFilters outcome crud-tests.test[0].y: expected an array, got {"$[i]":{"b":2}} +bulkWrite-arrayFilters.json FAIL BulkWrite updateMany with arrayFilters outcome crud-tests.test[0].y: expected an array, got {"$[i]":{"b":2}} +bulkWrite-arrayFilters.json FAIL BulkWrite with arrayFilters outcome crud-tests.test[0].y: expected an array, got {"$[i]":{"b":2}} +bulkWrite-collation.json FAIL BulkWrite with delete operations and collation bulkWrite.deletedCount: expected 4, got 0 +bulkWrite-collation.json FAIL BulkWrite with update operations and collation MongoBulkWriteError: internal error +bulkWrite-comment.json FAIL BulkWrite with string comment MongoBulkWriteError: bad update +bulkWrite-comment.json FAIL BulkWrite with document comment MongoBulkWriteError: bad update +bulkWrite-comment.json SKIP BulkWrite with comment - pre 4.4 needs server <= 4.2.99 +bulkWrite-delete-hint-serverError.json SKIP * needs server <= 4.3.3 +bulkWrite-deleteMany-hint-unacknowledged.json SKIP Unacknowledged deleteMany with hint string fails with client-side error on pre-4.4 server needs server <= 4.2.99 +bulkWrite-deleteMany-hint-unacknowledged.json SKIP Unacknowledged deleteMany with hint document fails with client-side error on pre-4.4 server needs server <= 4.2.99 +bulkWrite-deleteMany-hint-unacknowledged.json FAIL Unacknowledged deleteMany with hint string on 4.4+ server bulkWrite: unexpected extra keys ["insertedCount","matchedCount","modifiedCount","deletedCount","upsertedCount","upsertedIds","insertedIds"] +bulkWrite-deleteMany-hint-unacknowledged.json FAIL Unacknowledged deleteMany with hint document on 4.4+ server bulkWrite: unexpected extra keys ["insertedCount","matchedCount","modifiedCount","deletedCount","upsertedCount","upsertedIds","insertedIds"] +bulkWrite-deleteMany-let.json SKIP BulkWrite deleteMany with let option needs server >= 5.0 +bulkWrite-deleteMany-let.json FAIL BulkWrite deleteMany with let option unsupported (server-side error) bulkWrite: expected an error, the operation succeeded +bulkWrite-deleteMany-rawdata.json SKIP BulkWrite deleteMany with rawData option needs server >= 8.2.0 +bulkWrite-deleteOne-hint-unacknowledged.json SKIP Unacknowledged deleteOne with hint string fails with client-side error on pre-4.4 server needs server <= 4.2.99 +bulkWrite-deleteOne-hint-unacknowledged.json SKIP Unacknowledged deleteOne with hint document fails with client-side error on pre-4.4 server needs server <= 4.2.99 +bulkWrite-deleteOne-hint-unacknowledged.json FAIL Unacknowledged deleteOne with hint string on 4.4+ server bulkWrite: unexpected extra keys ["insertedCount","matchedCount","modifiedCount","deletedCount","upsertedCount","upsertedIds","insertedIds"] +bulkWrite-deleteOne-hint-unacknowledged.json FAIL Unacknowledged deleteOne with hint document on 4.4+ server bulkWrite: unexpected extra keys ["insertedCount","matchedCount","modifiedCount","deletedCount","upsertedCount","upsertedIds","insertedIds"] +bulkWrite-deleteOne-let.json SKIP BulkWrite deleteOne with let option needs server >= 5.0 +bulkWrite-deleteOne-let.json FAIL BulkWrite deleteOne with let option unsupported (server-side error) bulkWrite: expected an error, the operation succeeded +bulkWrite-deleteOne-rawdata.json SKIP BulkWrite deleteOne with rawData option needs server >= 8.2.0 +bulkWrite-errorResponse.json SKIP bulkWrite operations support errorResponse assertions runner: failPoint +bulkWrite-insertOne-dots_and_dollars.json SKIP Inserting document with top-level dollar-prefixed key on 5.0+ server needs server >= 5.0 +bulkWrite-insertOne-dots_and_dollars.json FAIL Inserting document with top-level dollar-prefixed key on pre-5.0 server yields server-side error bulkWrite: expected an error, the operation succeeded +bulkWrite-replaceOne-dots_and_dollars.json FAIL Replacing document with top-level dotted key on 3.6+ server MongoBulkWriteError: bad update +bulkWrite-replaceOne-dots_and_dollars.json SKIP Replacing document with dollar-prefixed key in embedded doc on 5.0+ server needs server >= 5.0 +bulkWrite-replaceOne-dots_and_dollars.json FAIL Replacing document with dotted key in embedded doc on 3.6+ server MongoBulkWriteError: bad update +bulkWrite-replaceOne-hint-unacknowledged.json FAIL Unacknowledged replaceOne with hint string on 4.2+ server MongoBulkWriteError: bad update +bulkWrite-replaceOne-hint-unacknowledged.json FAIL Unacknowledged replaceOne with hint document on 4.2+ server MongoBulkWriteError: bad update +bulkWrite-replaceOne-let.json SKIP BulkWrite replaceOne with let option needs server >= 5.0 +bulkWrite-replaceOne-let.json FAIL BulkWrite replaceOne with let option unsupported (server-side error) bulkWrite: expected an error, the operation succeeded +bulkWrite-replaceOne-rawdata.json SKIP BulkWrite replaceOne with rawData option needs server >= 8.2.0 +bulkWrite-replaceOne-sort.json SKIP BulkWrite replaceOne with sort option needs server >= 8.0 +bulkWrite-update-hint.json FAIL BulkWrite replaceOne with update hints MongoBulkWriteError: bad update +bulkWrite-updateMany-dots_and_dollars.json SKIP Updating document to set top-level dollar-prefixed key on 5.0+ server needs server >= 5.0 +bulkWrite-updateMany-dots_and_dollars.json SKIP Updating document to set top-level dotted key on 5.0+ server needs server >= 5.0 +bulkWrite-updateMany-dots_and_dollars.json SKIP Updating document to set dollar-prefixed key in embedded doc on 5.0+ server needs server >= 5.0 +bulkWrite-updateMany-dots_and_dollars.json SKIP Updating document to set dotted key in embedded doc on 5.0+ server needs server >= 5.0 +bulkWrite-updateMany-hint-unacknowledged.json FAIL Unacknowledged updateMany with hint string on 4.2+ server bulkWrite: unexpected extra keys ["insertedCount","matchedCount","modifiedCount","deletedCount","upsertedCount","upsertedIds","insertedIds"] +bulkWrite-updateMany-hint-unacknowledged.json FAIL Unacknowledged updateMany with hint document on 4.2+ server bulkWrite: unexpected extra keys ["insertedCount","matchedCount","modifiedCount","deletedCount","upsertedCount","upsertedIds","insertedIds"] +bulkWrite-updateMany-let.json SKIP BulkWrite updateMany with let option needs server >= 5.0 +bulkWrite-updateMany-let.json FAIL BulkWrite updateMany with let option unsupported (server-side error) bulkWrite: error message "update spec requires u" does not contain "'update.let' is an unknown field" +bulkWrite-updateMany-pipeline.json FAIL UpdateMany in bulk write using pipelines MongoBulkWriteError: update spec requires u +bulkWrite-updateMany-rawdata.json SKIP BulkWrite updateMany with rawData option needs server >= 8.2.0 +bulkWrite-updateMany-rawdata.json FAIL BulkWrite updateMany with rawData option on less than 8.2.0 - ignore argument MongoBulkWriteError: update spec requires u +bulkWrite-updateOne-dots_and_dollars.json SKIP Updating document to set top-level dollar-prefixed key on 5.0+ server needs server >= 5.0 +bulkWrite-updateOne-dots_and_dollars.json SKIP Updating document to set top-level dotted key on 5.0+ server needs server >= 5.0 +bulkWrite-updateOne-dots_and_dollars.json SKIP Updating document to set dollar-prefixed key in embedded doc on 5.0+ server needs server >= 5.0 +bulkWrite-updateOne-dots_and_dollars.json SKIP Updating document to set dotted key in embedded doc on 5.0+ server needs server >= 5.0 +bulkWrite-updateOne-hint-unacknowledged.json FAIL Unacknowledged updateOne with hint string on 4.2+ server bulkWrite: unexpected extra keys ["insertedCount","matchedCount","modifiedCount","deletedCount","upsertedCount","upsertedIds","insertedIds"] +bulkWrite-updateOne-hint-unacknowledged.json FAIL Unacknowledged updateOne with hint document on 4.2+ server bulkWrite: unexpected extra keys ["insertedCount","matchedCount","modifiedCount","deletedCount","upsertedCount","upsertedIds","insertedIds"] +bulkWrite-updateOne-let.json SKIP BulkWrite updateOne with let option needs server >= 5.0 +bulkWrite-updateOne-let.json FAIL BulkWrite updateOne with let option unsupported (server-side error) bulkWrite: error message "update spec requires u" does not contain "'update.let' is an unknown field" +bulkWrite-updateOne-pipeline.json FAIL UpdateOne in bulk write using pipelines MongoBulkWriteError: update spec requires u +bulkWrite-updateOne-rawdata.json SKIP BulkWrite updateOne with rawData option needs server >= 8.2.0 +bulkWrite-updateOne-rawdata.json FAIL BulkWrite updateOne with rawData option on less than 8.2.0 - ignore argument MongoBulkWriteError: update spec requires u +bulkWrite-updateOne-sort.json SKIP BulkWrite updateOne with sort option needs server >= 8.0 +bulkWrite.json FAIL BulkWrite with replaceOne operations MongoBulkWriteError: bad update +bulkWrite.json FAIL BulkWrite with updateOne operations bulkWrite.modifiedCount: expected 1, got 2 +bulkWrite.json FAIL BulkWrite with updateMany operations bulkWrite.modifiedCount: expected 2, got 4 +bulkWrite.json FAIL BulkWrite with mixed ordered operations MongoBulkWriteError: internal error +bulkWrite.json FAIL BulkWrite with mixed unordered operations MongoBulkWriteError: internal error +bypassDocumentValidation.json FAIL Aggregate with $out passes bypassDocumentValidation: false MongoServerError: Unrecognized pipeline stage name: '$out' +bypassDocumentValidation.json FAIL FindOneAndReplace passes bypassDocumentValidation: false MongoServerError: bad update +bypassDocumentValidation.json FAIL ReplaceOne passes bypassDocumentValidation: false MongoServerError: bad update +client-bulkWrite-delete-options.json SKIP * needs server >= 8.0 +client-bulkWrite-delete-rawdata.json SKIP client bulk write delete with rawData option needs server >= 8.2.0 +client-bulkWrite-delete-rawdata.json SKIP client bulk write delete with rawData option on less than 8.2.0 - ignore argument needs server >= 8.0 +client-bulkWrite-errorResponse.json SKIP * needs server >= 8.0 +client-bulkWrite-errors.json SKIP * needs server >= 8.0 +client-bulkWrite-mixed-namespaces.json SKIP * needs server >= 8.0 +client-bulkWrite-options.json SKIP * needs server >= 8.0 +client-bulkWrite-ordered.json SKIP * needs server >= 8.0 +client-bulkWrite-partialResults.json SKIP * needs server >= 8.0 +client-bulkWrite-replaceOne-rawdata.json SKIP client bulkWrite replaceOne with rawData option needs server >= 8.2.0 +client-bulkWrite-replaceOne-rawdata.json SKIP client bulk write replaceOne with rawData option on less than 8.2.0 - ignore argument needs server >= 8.0 +client-bulkWrite-replaceOne-sort.json SKIP * needs server >= 8.0 +client-bulkWrite-results.json SKIP * needs server >= 8.0 +client-bulkWrite-update-options.json SKIP * needs server >= 8.0 +client-bulkWrite-update-pipeline.json SKIP * needs server >= 8.0 +client-bulkWrite-update-rawdata.json SKIP client bulk write update with rawData option needs server >= 8.2.0 +client-bulkWrite-update-rawdata.json SKIP client bulk write update with rawData option on less than 8.2.0 - ignore argument needs server >= 8.0 +client-bulkWrite-update-validation.json SKIP client bulkWrite replaceOne prohibits atomic modifiers runner: operation clientBulkWrite +client-bulkWrite-update-validation.json SKIP client bulkWrite updateOne requires atomic modifiers runner: operation clientBulkWrite +client-bulkWrite-update-validation.json SKIP client bulkWrite updateMany requires atomic modifiers runner: operation clientBulkWrite +client-bulkWrite-updateOne-sort.json SKIP * needs server >= 8.0 +count-collation.json SKIP Deprecated count with collation runner: operation count +count-empty.json SKIP Deprecated count with empty collection runner: operation count +count-rawdata.json SKIP Deprecated count with rawData option needs server >= 8.2.0 +count-rawdata.json SKIP Deprecated count with rawData option on less than 8.2.0 - ignore argument runner: operation count +count.json FAIL Count documents with skip and limit countDocuments: expected 2, got 3 +count.json SKIP Deprecated count without a filter runner: operation count +count.json SKIP Deprecated count with a filter runner: operation count +count.json SKIP Deprecated count with skip and limit runner: operation count +countDocuments-comment.json SKIP countDocuments with document comment on less than 4.4.0 - server error needs server <= 4.3.99 +countDocuments-rawdata.json SKIP Count documents with rawData option needs server >= 8.2.0 +create-null-ids.json FAIL inserting _id with type null via insertOne countDocuments: expected 1, got 0 +create-null-ids.json FAIL inserting _id with type null via insertMany countDocuments: expected 1, got 0 +create-null-ids.json FAIL inserting _id with type null via updateOne countDocuments: expected 1, got 0 +create-null-ids.json FAIL inserting _id with type null via updateMany countDocuments: expected 1, got 0 +create-null-ids.json FAIL inserting _id with type null via replaceOne MongoServerError: internal error +create-null-ids.json FAIL inserting _id with type null via bulkWrite countDocuments: expected 1, got 0 +create-null-ids.json SKIP inserting _id with type null via clientBulkWrite needs server >= 8.0 +db-aggregate-rawdata.json SKIP Aggregate with rawData option needs server >= 8.2.0, needs topology replicaset +db-aggregate-rawdata.json FAIL Aggregate with rawData option on less than 8.2.0 - ignore argument MongoServerError: command requires a collection name +db-aggregate-write-readPreference.json SKIP * needs topology replicaset +db-aggregate.json FAIL Aggregate with $listLocalSessions MongoServerError: command requires a collection name +db-aggregate.json FAIL Aggregate with $listLocalSessions and allowDiskUse MongoServerError: command requires a collection name +deleteMany-collation.json FAIL DeleteMany when many documents match with collation deleteMany.deletedCount: expected 2, got 0 +deleteMany-comment.json SKIP deleteMany with comment - pre 4.4 needs server <= 4.2.99 +deleteMany-hint-serverError.json SKIP * needs server <= 4.3.3 +deleteMany-hint-unacknowledged.json SKIP Unacknowledged deleteMany with hint string fails with client-side error on pre-4.4 server needs server <= 4.2.99 +deleteMany-hint-unacknowledged.json SKIP Unacknowledged deleteMany with hint document fails with client-side error on pre-4.4 server needs server <= 4.2.99 +deleteMany-hint-unacknowledged.json FAIL Unacknowledged deleteMany with hint string on 4.4+ server deleteMany: unexpected extra keys ["deletedCount"] +deleteMany-hint-unacknowledged.json FAIL Unacknowledged deleteMany with hint document on 4.4+ server deleteMany: unexpected extra keys ["deletedCount"] +deleteMany-let.json SKIP deleteMany with let option needs server >= 5.0 +deleteMany-let.json FAIL deleteMany with let option unsupported (server-side error) deleteMany: expected an error, the operation succeeded +deleteMany-rawdata.json SKIP deleteMany with rawData option needs server >= 8.2.0 +deleteOne-collation.json FAIL DeleteOne when many documents matches with collation deleteOne.deletedCount: expected 1, got 0 +deleteOne-comment.json SKIP deleteOne with comment - pre 4.4 needs server <= 4.2.99 +deleteOne-errorResponse.json SKIP delete operations support errorResponse assertions runner: failPoint +deleteOne-hint-serverError.json SKIP * needs server <= 4.3.3 +deleteOne-hint-unacknowledged.json SKIP Unacknowledged deleteOne with hint string fails with client-side error on pre-4.4 server needs server <= 4.2.99 +deleteOne-hint-unacknowledged.json SKIP Unacknowledged deleteOne with hint document fails with client-side error on pre-4.4 server needs server <= 4.2.99 +deleteOne-hint-unacknowledged.json FAIL Unacknowledged deleteOne with hint string on 4.4+ server deleteOne: unexpected extra keys ["deletedCount"] +deleteOne-hint-unacknowledged.json FAIL Unacknowledged deleteOne with hint document on 4.4+ server deleteOne: unexpected extra keys ["deletedCount"] +deleteOne-let.json SKIP deleteOne with let option needs server >= 5.0 +deleteOne-let.json FAIL deleteOne with let option unsupported (server-side error) deleteOne: expected an error, the operation succeeded +deleteOne-rawdata.json SKIP deleteOne with rawData option needs server >= 8.2.0 +distinct-collation.json FAIL Distinct with a collation MongoServerError: no such command: 'distinct' +distinct-comment.json SKIP distinct with document comment needs server >= 4.4.14 +distinct-comment.json FAIL distinct with string comment MongoServerError: no such command: 'distinct' +distinct-hint.json SKIP * needs server >= 7.1.0 +distinct-rawdata.json SKIP distinct with rawData option needs server >= 8.2.0 +distinct-rawdata.json FAIL distinct with rawData option on less than 8.2.0 - ignore argument MongoServerError: no such command: 'distinct' +distinct.json FAIL Distinct without a filter MongoServerError: no such command: 'distinct' +distinct.json FAIL Distinct with a filter MongoServerError: no such command: 'distinct' +estimatedDocumentCount-comment.json SKIP estimatedDocumentCount with document comment needs server >= 4.4.14 +estimatedDocumentCount-comment.json FAIL estimatedDocumentCount with document comment - pre 4.4.14, server error estimatedDocumentCount: expected an error, the operation succeeded +estimatedDocumentCount-rawdata.json SKIP Estimated document count with rawData option needs server >= 8.2.0 +estimatedDocumentCount.json SKIP estimatedDocumentCount errors correctly--command error runner: failPoint +estimatedDocumentCount.json SKIP estimatedDocumentCount errors correctly--socket error runner: failPoint +estimatedDocumentCount.json FAIL estimatedDocumentCount works correctly on views estimatedDocumentCount: expected 2, got 0 +find-allowdiskuse-serverError.json SKIP * needs server <= 4.3.0 +find-collation.json FAIL Find with a collation find: expected 1 elements, got 0 +find-comment.json FAIL find with string comment find[0]: unexpected extra keys ["x"] +find-comment.json FAIL find with document comment find[0]: unexpected extra keys ["x"] +find-comment.json SKIP find with document comment - pre 4.4 needs server <= 4.2.99 +find-comment.json SKIP find with comment does not set comment on getMore - pre 4.4 needs server <= 4.3.99 +find-let.json SKIP Find with let option needs server >= 5.0 +find-let.json FAIL Find with let option unsupported (server-side error) find: expected an error, the operation succeeded +find-rawdata.json SKIP Find with rawData option needs server >= 8.2.0 +find.json FAIL Find with filter, sort, skip, and limit find: expected 2 elements, got 4 +find.json FAIL Find with limit, sort, and batchsize find: expected 4 elements, got 6 +find.json FAIL Find with batchSize equal to limit find: expected 4 elements, got 5 +findOne.json FAIL FindOne with filter, sort, and skip findOne._id: expected 5, got 3 +findOneAndDelete-collation.json FAIL FindOneAndDelete when one document matches with collation findOneAndDelete: expected a document, got null +findOneAndDelete-comment.json SKIP findOneAndDelete with comment - pre 4.4 needs server <= 4.2.99 +findOneAndDelete-hint-serverError.json SKIP * needs server <= 4.3.3 +findOneAndDelete-hint-unacknowledged.json SKIP Unacknowledged findOneAndDelete with hint string fails with client-side error on pre-4.4 server needs server <= 4.2.99 +findOneAndDelete-hint-unacknowledged.json SKIP Unacknowledged findOneAndDelete with hint document fails with client-side error on pre-4.4 server needs server <= 4.2.99 +findOneAndDelete-hint-unacknowledged.json FAIL Unacknowledged findOneAndDelete with hint string on 4.4+ server findOneAndDelete: expected null, got {"_id":2,"x":22} +findOneAndDelete-hint-unacknowledged.json FAIL Unacknowledged findOneAndDelete with hint document on 4.4+ server findOneAndDelete: expected null, got {"_id":2,"x":22} +findOneAndDelete-let.json SKIP findOneAndDelete with let option needs server >= 5.0 +findOneAndDelete-let.json FAIL findOneAndDelete with let option unsupported (server-side error) findOneAndDelete: expected an error, the operation succeeded +findOneAndDelete-rawdata.json SKIP findOneAndDelete with rawData option needs server >= 8.2.0 +findOneAndReplace-collation.json FAIL FindOneAndReplace when one document matches with collation returning the document after modification findOneAndReplace: expected a document, got null +findOneAndReplace-comment.json FAIL findOneAndReplace with string comment MongoServerError: bad update +findOneAndReplace-comment.json FAIL findOneAndReplace with document comment MongoServerError: bad update +findOneAndReplace-comment.json SKIP findOneAndReplace with comment - pre 4.4 needs server <= 4.2.99 +findOneAndReplace-dots_and_dollars.json FAIL Replacing document with top-level dotted key on 3.6+ server MongoServerError: bad update +findOneAndReplace-dots_and_dollars.json SKIP Replacing document with dollar-prefixed key in embedded doc on 5.0+ server needs server >= 5.0 +findOneAndReplace-dots_and_dollars.json FAIL Replacing document with dotted key in embedded doc on 3.6+ server MongoServerError: bad update +findOneAndReplace-hint-serverError.json SKIP * needs server <= 4.3.0 +findOneAndReplace-hint-unacknowledged.json SKIP Unacknowledged findOneAndReplace with hint string fails with client-side error on pre-4.4 server needs server <= 4.2.99 +findOneAndReplace-hint-unacknowledged.json SKIP Unacknowledged findOneAndReplace with hint document fails with client-side error on pre-4.4 server needs server <= 4.2.99 +findOneAndReplace-hint-unacknowledged.json FAIL Unacknowledged findOneAndReplace with hint string on 4.4+ server MongoServerError: bad update +findOneAndReplace-hint-unacknowledged.json FAIL Unacknowledged findOneAndReplace with hint document on 4.4+ server MongoServerError: bad update +findOneAndReplace-hint.json FAIL FindOneAndReplace with hint string MongoServerError: bad update +findOneAndReplace-hint.json FAIL FindOneAndReplace with hint document MongoServerError: bad update +findOneAndReplace-let.json SKIP findOneAndReplace with let option needs server >= 5.0 +findOneAndReplace-let.json FAIL findOneAndReplace with let option unsupported (server-side error) findOneAndReplace: expected an error, the operation succeeded +findOneAndReplace-rawdata.json SKIP findOneAndReplace with rawData option needs server >= 8.2.0 +findOneAndReplace-rawdata.json FAIL findOneAndReplace with rawData option on less than 8.2.0 - ignore argument MongoServerError: bad update +findOneAndReplace-upsert.json FAIL FindOneAndReplace when no documents match without id specified with upsert returning the document before modification MongoServerError: internal error +findOneAndReplace-upsert.json FAIL FindOneAndReplace when no documents match without id specified with upsert returning the document after modification MongoServerError: internal error +findOneAndReplace-upsert.json FAIL FindOneAndReplace when no documents match with id specified with upsert returning the document before modification MongoServerError: internal error +findOneAndReplace-upsert.json FAIL FindOneAndReplace when no documents match with id specified with upsert returning the document after modification MongoServerError: internal error +findOneAndReplace.json FAIL FindOneAndReplace when many documents match returning the document before modification MongoServerError: bad update +findOneAndReplace.json FAIL FindOneAndReplace when many documents match returning the document after modification MongoServerError: bad update +findOneAndReplace.json FAIL FindOneAndReplace when one document matches returning the document before modification MongoServerError: bad update +findOneAndReplace.json FAIL FindOneAndReplace when one document matches returning the document after modification MongoServerError: bad update +findOneAndUpdate-arrayFilters.json FAIL FindOneAndUpdate when no document matches arrayFilters outcome crud-v1.coll[0].y: expected an array, got {"$[i]":{"b":2}} +findOneAndUpdate-arrayFilters.json FAIL FindOneAndUpdate when one document matches arrayFilters outcome crud-v1.coll[0].y: expected an array, got {"$[i]":{"b":2}} +findOneAndUpdate-arrayFilters.json FAIL FindOneAndUpdate when multiple documents match arrayFilters outcome crud-v1.coll[0].y: expected an array, got {"$[i]":{"b":2}} +findOneAndUpdate-collation.json FAIL FindOneAndUpdate when many documents match with collation returning the document before modification findOneAndUpdate: expected a document, got null +findOneAndUpdate-comment.json FAIL findOneAndUpdate with string comment MongoServerError: update must be a document +findOneAndUpdate-comment.json FAIL findOneAndUpdate with document comment MongoServerError: update must be a document +findOneAndUpdate-comment.json SKIP findOneAndUpdate with comment - pre 4.4 needs server <= 4.2.99 +findOneAndUpdate-dots_and_dollars.json SKIP Updating document to set top-level dollar-prefixed key on 5.0+ server needs server >= 5.0 +findOneAndUpdate-dots_and_dollars.json SKIP Updating document to set top-level dotted key on 5.0+ server needs server >= 5.0 +findOneAndUpdate-dots_and_dollars.json SKIP Updating document to set dollar-prefixed key in embedded doc on 5.0+ server needs server >= 5.0 +findOneAndUpdate-dots_and_dollars.json SKIP Updating document to set dotted key in embedded doc on 5.0+ server needs server >= 5.0 +findOneAndUpdate-errorResponse.json FAIL findOneAndUpdate DuplicateKey error is accessible findOneAndUpdate.errorResponse.keyPattern: missing from actual +findOneAndUpdate-errorResponse.json SKIP findOneAndUpdate document validation errInfo is accessible needs server >= 5.0 +findOneAndUpdate-hint-serverError.json SKIP * needs server <= 4.3.0 +findOneAndUpdate-hint-unacknowledged.json SKIP Unacknowledged findOneAndUpdate with hint string fails with client-side error on pre-4.4 server needs server <= 4.2.99 +findOneAndUpdate-hint-unacknowledged.json SKIP Unacknowledged findOneAndUpdate with hint document fails with client-side error on pre-4.4 server needs server <= 4.2.99 +findOneAndUpdate-hint-unacknowledged.json FAIL Unacknowledged findOneAndUpdate with hint string on 4.4+ server findOneAndUpdate: expected null, got {"_id":2,"x":22} +findOneAndUpdate-hint-unacknowledged.json FAIL Unacknowledged findOneAndUpdate with hint document on 4.4+ server findOneAndUpdate: expected null, got {"_id":2,"x":22} +findOneAndUpdate-let.json SKIP findOneAndUpdate with let option needs server >= 5.0 +findOneAndUpdate-let.json FAIL findOneAndUpdate with let option unsupported (server-side error) findOneAndUpdate: expected an error, the operation succeeded +findOneAndUpdate-pipeline.json FAIL FindOneAndUpdate using pipelines MongoServerError: update must be a document +findOneAndUpdate-rawdata.json SKIP findOneAndUpdate with rawData option needs server >= 8.2.0 +findOneAndUpdate-rawdata.json FAIL findOneAndUpdate with rawData option on less than 8.2.0 - ignore argument MongoServerError: update must be a document +findOneAndUpdate.json FAIL FindOneAndUpdate when many documents match returning the document after modification findOneAndUpdate.x: expected 23, got 22 +findOneAndUpdate.json FAIL FindOneAndUpdate when one document matches returning the document after modification findOneAndUpdate.x: expected 23, got 22 +findOneAndUpdate.json FAIL FindOneAndUpdate when no documents match with upsert returning the document after modification findOneAndUpdate: expected a document, got null +insertMany-comment.json SKIP insertMany with comment - pre 4.4 needs server <= 4.2.99 +insertMany-dots_and_dollars.json SKIP Inserting document with top-level dollar-prefixed key on 5.0+ server needs server >= 5.0 +insertMany-dots_and_dollars.json FAIL Inserting document with top-level dollar-prefixed key on pre-5.0 server yields server-side error insertMany: expected an error, the operation succeeded +insertMany-dots_and_dollars.json FAIL Inserting document with top-level dotted key insertMany: unexpected extra keys ["insertedCount"] +insertMany-dots_and_dollars.json FAIL Inserting document with dollar-prefixed key in embedded doc insertMany: unexpected extra keys ["insertedCount"] +insertMany-dots_and_dollars.json FAIL Inserting document with dotted key in embedded doc insertMany: unexpected extra keys ["insertedCount"] +insertMany-rawdata.json SKIP insertMany with rawData option needs server >= 8.2.0 +insertMany.json FAIL InsertMany with non-existing documents insertMany: unexpected extra keys ["insertedCount"] +insertOne-comment.json SKIP insertOne with comment - pre 4.4 needs server <= 4.2.99 +insertOne-dots_and_dollars.json SKIP Inserting document with top-level dollar-prefixed key on 5.0+ server needs server >= 5.0 +insertOne-dots_and_dollars.json FAIL Inserting document with top-level dollar-prefixed key on pre-5.0 server yields server-side error insertOne: expected an error, the operation succeeded +insertOne-dots_and_dollars.json FAIL Inserting document with dollar-prefixed key in _id yields server-side error insertOne: expected an error, the operation succeeded +insertOne-dots_and_dollars.json FAIL Unacknowledged write using dollar-prefixed or dotted keys may be silently rejected on pre-5.0 server insertOne: unexpected extra keys ["insertedId"] +insertOne-errorResponse.json SKIP insert operations support errorResponse assertions runner: failPoint +insertOne-rawdata.json SKIP insertOne with rawData option needs server >= 8.2.0 +replaceOne-collation.json FAIL ReplaceOne when one document matches with collation replaceOne.matchedCount: expected 1, got 0 +replaceOne-comment.json FAIL ReplaceOne with string comment MongoServerError: bad update +replaceOne-comment.json FAIL ReplaceOne with document comment MongoServerError: bad update +replaceOne-comment.json SKIP ReplaceOne with comment - pre 4.4 needs server <= 4.2.99 +replaceOne-dots_and_dollars.json FAIL Replacing document with top-level dotted key on 3.6+ server MongoServerError: bad update +replaceOne-dots_and_dollars.json SKIP Replacing document with dollar-prefixed key in embedded doc on 5.0+ server needs server >= 5.0 +replaceOne-dots_and_dollars.json FAIL Replacing document with dotted key in embedded doc on 3.6+ server MongoServerError: bad update +replaceOne-hint-unacknowledged.json FAIL Unacknowledged replaceOne with hint string on 4.2+ server MongoServerError: bad update +replaceOne-hint-unacknowledged.json FAIL Unacknowledged replaceOne with hint document on 4.2+ server MongoServerError: bad update +replaceOne-hint.json FAIL ReplaceOne with hint string MongoServerError: bad update +replaceOne-hint.json FAIL ReplaceOne with hint document MongoServerError: bad update +replaceOne-let.json SKIP ReplaceOne with let option needs server >= 5.0 +replaceOne-let.json FAIL ReplaceOne with let option unsupported (server-side error) replaceOne: expected an error, the operation succeeded +replaceOne-rawdata.json SKIP ReplaceOne with rawData option needs server >= 8.2.0 +replaceOne-rawdata.json FAIL ReplaceOne with rawData option on less than 8.2.0 - ignore argument MongoServerError: bad update +replaceOne-sort.json SKIP ReplaceOne with sort option needs server >= 8.0 +replaceOne.json FAIL ReplaceOne when many documents match MongoServerError: bad update +replaceOne.json FAIL ReplaceOne when one document matches MongoServerError: bad update +replaceOne.json FAIL ReplaceOne with upsert when no documents match without an id specified MongoServerError: internal error +replaceOne.json FAIL ReplaceOne with upsert when no documents match with an id specified MongoServerError: internal error +updateMany-arrayFilters.json FAIL UpdateMany when no documents match arrayFilters updateMany.modifiedCount: expected 0, got 2 +updateMany-arrayFilters.json FAIL UpdateMany when one document matches arrayFilters updateMany.modifiedCount: expected 1, got 2 +updateMany-arrayFilters.json FAIL UpdateMany when multiple documents match arrayFilters outcome crud-v1.coll[0].y: expected an array, got {"$[i]":{"b":2}} +updateMany-collation.json FAIL UpdateMany when many documents match with collation updateMany.matchedCount: expected 2, got 1 +updateMany-comment.json SKIP UpdateMany with comment - pre 4.4 needs server <= 4.2.99 +updateMany-dots_and_dollars.json SKIP Updating document to set top-level dollar-prefixed key on 5.0+ server needs server >= 5.0 +updateMany-dots_and_dollars.json SKIP Updating document to set top-level dotted key on 5.0+ server needs server >= 5.0 +updateMany-dots_and_dollars.json SKIP Updating document to set dollar-prefixed key in embedded doc on 5.0+ server needs server >= 5.0 +updateMany-dots_and_dollars.json SKIP Updating document to set dotted key in embedded doc on 5.0+ server needs server >= 5.0 +updateMany-hint-unacknowledged.json FAIL Unacknowledged updateMany with hint string on 4.2+ server updateMany: unexpected extra keys ["modifiedCount","upsertedId","upsertedCount","matchedCount"] +updateMany-hint-unacknowledged.json FAIL Unacknowledged updateMany with hint document on 4.2+ server updateMany: unexpected extra keys ["modifiedCount","upsertedId","upsertedCount","matchedCount"] +updateMany-let.json SKIP updateMany with let option needs server >= 5.0 +updateMany-let.json FAIL updateMany with let option unsupported (server-side error) updateMany: error message "update spec requires u" does not contain "'update.let' is an unknown field" +updateMany-pipeline.json FAIL UpdateMany using pipelines MongoServerError: update spec requires u +updateMany-rawdata.json SKIP updateMany with rawData option needs server >= 8.2.0 +updateOne-arrayFilters.json FAIL UpdateOne when no document matches arrayFilters updateOne.modifiedCount: expected 0, got 1 +updateOne-arrayFilters.json FAIL UpdateOne when one document matches arrayFilters outcome crud-v1.coll[0].y: expected an array, got {"$[i]":{"b":2}} +updateOne-arrayFilters.json FAIL UpdateOne when multiple documents match arrayFilters outcome crud-v1.coll[0].y: expected an array, got {"$[i]":{"b":2}} +updateOne-arrayFilters.json FAIL UpdateOne when no documents match multiple arrayFilters updateOne.modifiedCount: expected 0, got 1 +updateOne-arrayFilters.json FAIL UpdateOne when one document matches multiple arrayFilters outcome crud-v1.coll[2].y: expected an array, got {"$[i]":{"c":{"$[j]":{"d":0}}}} +updateOne-collation.json FAIL UpdateOne when one document matches with collation updateOne.matchedCount: expected 1, got 0 +updateOne-comment.json SKIP UpdateOne with comment - pre 4.4 needs server <= 4.2.99 +updateOne-dots_and_dollars.json SKIP Updating document to set top-level dollar-prefixed key on 5.0+ server needs server >= 5.0 +updateOne-dots_and_dollars.json SKIP Updating document to set top-level dotted key on 5.0+ server needs server >= 5.0 +updateOne-dots_and_dollars.json SKIP Updating document to set dollar-prefixed key in embedded doc on 5.0+ server needs server >= 5.0 +updateOne-dots_and_dollars.json SKIP Updating document to set dotted key in embedded doc on 5.0+ server needs server >= 5.0 +updateOne-errorResponse.json SKIP update operations support errorResponse assertions runner: failPoint +updateOne-hint-unacknowledged.json FAIL Unacknowledged updateOne with hint string on 4.2+ server updateOne: unexpected extra keys ["modifiedCount","upsertedId","upsertedCount","matchedCount"] +updateOne-hint-unacknowledged.json FAIL Unacknowledged updateOne with hint document on 4.2+ server updateOne: unexpected extra keys ["modifiedCount","upsertedId","upsertedCount","matchedCount"] +updateOne-let.json SKIP UpdateOne with let option needs server >= 5.0 +updateOne-let.json FAIL UpdateOne with let option unsupported (server-side error) updateOne: error message "update spec requires u" does not contain "'update.let' is an unknown field" +updateOne-pipeline.json FAIL UpdateOne using pipelines MongoServerError: update spec requires u +updateOne-rawdata.json SKIP UpdateOne with rawData option needs server >= 8.2.0 +updateOne-sort.json SKIP UpdateOne with sort option needs server >= 8.0 +updateOne-sort.json FAIL updateOne with sort option unsupported (server-side error) updateOne: expected an error, the operation succeeded