tests/spec: MongoDB spec-test runner and the M0 scorecard
PLAN D2 makes the official specification suites the gate for command semantics; D7.6 asks for the harness to exist at M0 with a recorded baseline. This is that harness, pinned on both sides -- mongodb/specifications @ 615e0f9 and mongodb@7.5.0 -- because a scorecard is only comparable across milestones if a delta cannot be an upstream test change. It implements the unified format's Evaluating Matches algorithm as written, including the two rules that decide whether a pass is earned: extra keys are tolerated only in a root document, and numeric types compare flexibly. Anything unimplemented is a SKIP with a reason, never a pass, and the one assertion class not yet checked -- expectEvents, i.e. command monitoring -- is disclosed at the top of the scorecard so `pass` reads as an upper bound. First honest run: 131 pass, 161 fail, 195 skip over 175 files, zero timeouts. Getting there took four attempts, and the failures are documented in the README because each would have shipped a scorecard claiming a compatibility gap that did not exist. Two were genuine leaks in this runner (clients left open when a case timed out; clients registered for cleanup only after `await connect()`, plus abandoned cases still creating more). The third I misdiagnosed as machine load. The fourth attempt found the real cause: a leaked catalog lock in the engine, fixed separately, which alone accounts for the jump from 45 passes to 131. So the runner carries its own guards: per-operation CSOT timeouts so work is never abandoned, an active-handle census per file, an end-of-run tripwire for stray timers, a hard stop if the server dies rather than emitting hundreds of misleading ECONNREFUSED failures, and --skip/--limit for bisecting a run whose failures depend on position. The README states the rule plainly -- a long unbroken tail of timeouts is a harness bug until proven otherwise -- and the two commands that settle it. Also fixes bench-run.sh, which copied its report over bench-latest.txt unconditionally, including after a run that only warned -- so a degraded run could silently replace the baseline that PLAN D7.5 makes a milestone gate.
This commit is contained in:
2
.gitignore
vendored
2
.gitignore
vendored
@@ -2,3 +2,5 @@
|
||||
zig-out/
|
||||
*.log
|
||||
node_modules/
|
||||
# Pinned upstream spec suites, fetched by tests/spec/fetch.sh (PLAN D2).
|
||||
tests/spec/specifications/
|
||||
|
||||
52
PLAN.md
52
PLAN.md
@@ -380,11 +380,36 @@ stay in the arena — it outlives the command; it is only the ArrayList's own
|
||||
buffer whose allocator has to match its `deinit`). Add an e2e case for a bare
|
||||
`$sort` pipeline, and mutation-check it by restoring `arena` on the append.
|
||||
|
||||
### Bug found by the spec harness: a nameless command leaked the catalog lock
|
||||
|
||||
**Fixed. Severity: permanent denial of service, remotely triggerable by an
|
||||
ordinary query.** This one blocked the M0 scorecard outright and cost three
|
||||
invalid baselines.
|
||||
|
||||
`dispatch` resolved the namespace *after* taking the catalog lock and bailed
|
||||
with `orelse return` when the collection name was missing. A plain return runs
|
||||
neither the `errdefer` nor the explicit unlocks, so the lock was held — shared —
|
||||
for the life of the process. `db.aggregate(...)` reaches it: that sends
|
||||
`{aggregate: 1}`, whose value is not a string.
|
||||
|
||||
Worth understanding how it hid, because the shape recurs: a leaked *shared* lock
|
||||
is invisible to readers. `ping` and `listDatabases` kept answering in
|
||||
microseconds and an external prober saw `ok 15ms` straight through the hang, so
|
||||
the server looked healthy. Only a write needing the catalog exclusive to create
|
||||
a collection blocked — so the damage appeared one command later, on a different
|
||||
connection, as a client-side timeout with nothing linking it to the cause.
|
||||
|
||||
What actually found it: the driver's own command log, showing an insert sitting
|
||||
for exactly `socketTimeoutMS` against an idle engine.
|
||||
|
||||
Namespace resolution now happens before any lock is taken. Two lessons kept in
|
||||
`tests/spec/README.md`: a responsive server does not exonerate the engine, and
|
||||
probe with the operation that is stuck rather than with `ping`.
|
||||
|
||||
### Bug found by the spec harness: unacknowledged writes corrupt the connection
|
||||
|
||||
Recorded here rather than fixed in M0, since it is M1's surface — but it is a
|
||||
correctness bug, not a missing feature, and it is worth doing early because it
|
||||
is a handful of lines.
|
||||
**Fixed** (in M0 rather than M1 as originally recorded — it is a correctness
|
||||
bug, not a missing feature, and it is a handful of lines).
|
||||
|
||||
`wire.Message.flags` is parsed and stored but **never read**. A driver sending
|
||||
an unacknowledged write (`writeConcern: {w: 0}`) sets `moreToCome` (bit 0x2)
|
||||
@@ -403,9 +428,24 @@ next command on same connection FAILED: MongoUnexpectedServerResponseError:
|
||||
performance choice — breaks a connection on first use, and it is invisible to
|
||||
the existing e2e suites because none of them use it.
|
||||
|
||||
Fix: when `flags & 0x2` is set on an OP_MSG request, run the command and write
|
||||
no reply. Add an e2e case for it (unacknowledged write, then a read on the same
|
||||
connection), and mutation-check it by clearing the flag test.
|
||||
Fixed by suppressing the reply when `flags & 0x2` is set on an OP_MSG request:
|
||||
the command still runs. The e2e case pins `maxPoolSize` to 1, since with a
|
||||
larger pool the driver may hand the next operation a different connection and
|
||||
hide it.
|
||||
|
||||
### What the harness was worth
|
||||
|
||||
Its first honest run — after the two bugs above and two genuine leaks in the
|
||||
runner itself — reports **131 pass, 161 fail, 195 skip** over 175 files, with
|
||||
zero timeouts. Before the catalog-lock fix the same suite reported 45 passes:
|
||||
that gap is the measure of what one leaked lock was hiding, and of why a
|
||||
scorecard is only worth committing once it disagrees with nothing that passes in
|
||||
isolation.
|
||||
|
||||
The remaining failures are real work, and they cluster usefully: update-operator
|
||||
gaps (`bad update`, `update must be a document` — M3), unimplemented commands
|
||||
(`distinct`, `$merge`, `$out` — M2/M3), and result-shape mismatches in
|
||||
`bulkWrite`/`insertMany`. That list, not the total, is the milestone backlog.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -57,6 +57,7 @@ for C in $CLIENTS; do
|
||||
MD_D=$(echo "$MD" | sed -E 's/.*\t([0-9.]+) docs\/s.*/\1/')
|
||||
if [ -z "$MFDB_D" ] || [ -z "$MD_D" ]; then
|
||||
echo "WARNING: concurrent run at $C clients produced no result (mfdb='$MFDB' md='$MD')" >&2
|
||||
DEGRADED=1
|
||||
fi
|
||||
RATIO=$(node -e "const m=Number('$MFDB_D'),d=Number('$MD_D');console.log(d>0?(m/d).toFixed(1)+'x':'—')")
|
||||
echo "clients $C $MFDB_D $MD_D $RATIO" | tee -a "$CC"
|
||||
@@ -148,5 +149,17 @@ if [ -f "$LATEST" ]; then
|
||||
else
|
||||
echo "(no previous run to diff — this is the baseline)"
|
||||
fi
|
||||
# bench-latest.txt is the baseline the next run diffs against, and PLAN D7.5
|
||||
# makes "no phase8 row regresses" a milestone gate -- so a run that only
|
||||
# half-produced its numbers must not become the thing we compare to. This used
|
||||
# to be an unconditional copy, which meant a degraded run silently replaced the
|
||||
# baseline it was supposed to be measured against, and the missing rows then
|
||||
# read as "no change" forever.
|
||||
if [ "${DEGRADED:-0}" = "1" ]; then
|
||||
echo >&2
|
||||
echo "NOT updating $LATEST: this run was degraded (see WARNING above)." >&2
|
||||
echo "The report is still at $REPORT if you want it." >&2
|
||||
exit 1
|
||||
fi
|
||||
cp "$REPORT" "$LATEST"
|
||||
echo; echo "latest: $LATEST"
|
||||
|
||||
136
tests/spec/README.md
Normal file
136
tests/spec/README.md
Normal file
@@ -0,0 +1,136 @@
|
||||
# MongoDB spec tests
|
||||
|
||||
PLAN D2 makes the official MongoDB JSON specification suites the gate for
|
||||
command semantics: it turns "maximally compatible" into a concrete list of test
|
||||
files rather than a judgement call. This directory holds the runner and the
|
||||
committed scorecard.
|
||||
|
||||
```sh
|
||||
bash tests/spec/fetch.sh # pinned suites (~175 files, gitignored)
|
||||
zig build # the runner spawns this binary
|
||||
node tests/spec/run.js # run everything
|
||||
node tests/spec/run.js --scorecard # ... and rewrite scorecard.txt
|
||||
node tests/spec/run.js --file find.json --verbose
|
||||
node tests/spec/run.js --url mongodb://127.0.0.1:27020 # use a server you started
|
||||
```
|
||||
|
||||
## What is pinned, and why both halves matter
|
||||
|
||||
- **Suites**: `mongodb/specifications` @ `615e0f9`, in `fetch.sh`.
|
||||
- **Driver**: `mongodb@7.5.0`, via `tests/e2e/package-lock.json`.
|
||||
|
||||
A scorecard is only comparable across milestones if both are pinned — otherwise
|
||||
a delta could be an upstream test change rather than an engine change. Bump
|
||||
either one in its own commit and re-record the scorecard in that same commit.
|
||||
|
||||
The suites are fetched rather than vendored: they are someone else's corpus,
|
||||
upstream rewrites them wholesale, and a pinned commit gives the same
|
||||
reproducibility without putting them in this repo's history.
|
||||
|
||||
## Scope
|
||||
|
||||
`source/crud/tests/unified/` — 175 files. The aggregate tests live there too
|
||||
(`aggregate*.json`), so this one directory is PLAN M0's "crud + aggregate".
|
||||
|
||||
The runner implements the unified test format's **Evaluating Matches**
|
||||
algorithm as written in the spec, including the two rules that decide whether a
|
||||
result is a real pass:
|
||||
|
||||
- extra keys in the actual document are tolerated **only** in a root document;
|
||||
- numeric types (int32 / int64 / double) compare flexibly.
|
||||
|
||||
Supported: `client`/`database`/`collection` entities, `initialData`,
|
||||
`outcome`, `expectError` (code, codeName, contains, labels, errorResponse),
|
||||
`saveResultAsEntity`, `runOnRequirements` gating, and the `$$type`,
|
||||
`$$exists`, `$$unsetOrMatches`, `$$matchesEntity`, `$$matchesHexBytes`
|
||||
operators.
|
||||
|
||||
**Not asserted yet: `expectEvents`** (command monitoring). Those assertions are
|
||||
about the command shape the driver emits rather than result semantics. Ignoring
|
||||
them lets some cases pass that a complete runner would fail, so **treat `pass`
|
||||
as an upper bound** until M1 wires events up. This is stated again at the top of
|
||||
`scorecard.txt` so the number is never read out of context.
|
||||
|
||||
Not supported, each reported as SKIP with a reason and never as PASS: session
|
||||
and bucket entities (M4 / GridFS), `failPoint`, client-side encryption,
|
||||
`testRunner` operations, and any operation or matcher the runner does not know.
|
||||
|
||||
## Reading the scorecard
|
||||
|
||||
`scorecard.txt` records the totals, a per-file breakdown, and every
|
||||
non-passing case with its reason. The distinction that matters:
|
||||
|
||||
- **FAIL** — the engine answered, and answered differently from the spec. Real
|
||||
work. An operation that never answered inside `--op-timeout-ms` (default 3 s,
|
||||
enforced by the driver itself via CSOT `timeoutMS`) is also a FAIL, because
|
||||
"no answer" is a result. There is a second, much longer `--case-timeout-ms`
|
||||
backstop for a hang the driver cannot see; if it ever fires, treat the run
|
||||
with suspicion — see the trap below.
|
||||
- **SKIP** — nobody claims anything. Either the suite needs a feature whose
|
||||
milestone has not landed, or the runner does not implement it yet.
|
||||
|
||||
M0's gate (PLAN D7.6) is only that the harness exists and the baseline is
|
||||
recorded. **A red baseline is the expected state**, so `run.js` exits 0 as long
|
||||
as it ran; it is a measuring tool, not a pass/fail gate. Later milestones move
|
||||
the numbers, and each one commits the new scorecard (PLAN D9).
|
||||
|
||||
## A trap worth knowing about: the harness can invent failures
|
||||
|
||||
The first baseline attempt reported ~77 timeout FAILs that did not exist. Every
|
||||
case from one point onward timed out, while a `ping` from a separate process
|
||||
answered instantly — which read convincingly as a server-side wedge, and was
|
||||
not.
|
||||
|
||||
The cause was in this runner. `buildEntities` opened `MongoClient`s, and a case
|
||||
that timed out before it returned left them unclosed; each one keeps a
|
||||
connection pool and a heartbeat timer. Once enough accumulated, Node's event
|
||||
loop was starved badly enough that the per-case timer fired before operations
|
||||
could finish. Then every later case "failed".
|
||||
|
||||
Two things guard it now: per-test clients are owned by the caller and closed
|
||||
unconditionally, including on a partial failure; and the run ends by checking
|
||||
how many timers are still active, warning loudly if the answer is more than a
|
||||
handful.
|
||||
|
||||
The general rule, since it will come up again: **a run with a long unbroken tail
|
||||
of timeouts is a harness bug until proven otherwise.** Confirm it by running the
|
||||
first timing-out file on its own — if it passes in isolation, the failures are
|
||||
this runner's, not the engine's.
|
||||
|
||||
### ... but the third time it was the engine
|
||||
|
||||
A later attempt produced 166 timeout FAILs starting at file 70. I first blamed
|
||||
machine load — a concurrent `zig build test` against a then-3-second budget —
|
||||
and that was **wrong**. The evidence against it: the collapse reproduced on an
|
||||
idle machine, at the same file, with a 10 s budget.
|
||||
|
||||
The actual cause was a leaked catalog lock in the engine, and it is worth
|
||||
knowing how it hid. `db-aggregate.json` sends `{aggregate: 1}`, which names no
|
||||
collection; dispatch resolved the namespace after taking the catalog lock and
|
||||
bailed with a plain `return`, holding it shared forever. A leaked *shared* lock
|
||||
is invisible to readers, so the server stayed perfectly responsive — an external
|
||||
prober got `ok 15ms` right through the hang — and only the next write that had
|
||||
to take the catalog exclusive to create a collection blocked. The failure
|
||||
therefore surfaced one file later, on a different connection, as a client-side
|
||||
timeout with nothing pointing at its cause.
|
||||
|
||||
Two lessons for using this runner:
|
||||
|
||||
- **A healthy-looking server does not exonerate the engine.** Probe with the
|
||||
operation that is actually stuck, not with `ping`.
|
||||
- **The driver's own command log is the fastest way in.** It showed an insert
|
||||
sitting for exactly `socketTimeoutMS` against an idle engine, which is what
|
||||
turned a week-long-looking mystery into a five-line fix:
|
||||
|
||||
```sh
|
||||
MONGODB_LOG_COMMAND=debug MONGODB_LOG_PATH=stderr \
|
||||
node tests/spec/run.js --skip 68 --limit 2 2>drv.log
|
||||
```
|
||||
|
||||
`--skip`/`--limit` exist for exactly this: the collapse reduced to a
|
||||
reproducible two-file window, which is what made it tractable.
|
||||
|
||||
Still worth recording the baseline on an otherwise idle machine, and do not
|
||||
tighten `--op-timeout-ms` to make a run finish sooner — a tight budget turns
|
||||
load into apparent engine failures, which is how I misdiagnosed this once
|
||||
already.
|
||||
44
tests/spec/fetch.sh
Executable file
44
tests/spec/fetch.sh
Executable file
@@ -0,0 +1,44 @@
|
||||
#!/usr/bin/env bash
|
||||
# Fetch the pinned mongodb/specifications test suites.
|
||||
#
|
||||
# Pinned, not vendored: the suites are ~175 JSON files that upstream rewrites
|
||||
# wholesale, and a pinned commit gives the same reproducibility as vendoring
|
||||
# without putting someone else's test corpus in this repo's history. PLAN D2
|
||||
# requires the pin; `specifications/` is gitignored.
|
||||
#
|
||||
# Bump SPEC_COMMIT deliberately, in its own commit, and re-record the
|
||||
# scorecard in the same commit -- otherwise a scorecard delta cannot be told
|
||||
# apart from an upstream change.
|
||||
set -euo pipefail
|
||||
|
||||
SPEC_COMMIT="615e0f9ca5f554614636c098225fbbf1be55565d"
|
||||
SPEC_REPO="https://github.com/mongodb/specifications.git"
|
||||
|
||||
cd "$(dirname "$0")"
|
||||
DEST="specifications"
|
||||
|
||||
if [ -d "$DEST/.git" ] && [ "$(git -C "$DEST" rev-parse HEAD 2>/dev/null || true)" = "$SPEC_COMMIT" ]; then
|
||||
echo "specs already at $SPEC_COMMIT"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
rm -rf "$DEST"
|
||||
mkdir -p "$DEST"
|
||||
cd "$DEST"
|
||||
|
||||
# Sparse + depth 1: the full history is ~100 MB and we need two directories.
|
||||
git init -q .
|
||||
git remote add origin "$SPEC_REPO"
|
||||
git config core.sparseCheckout true
|
||||
mkdir -p .git/info
|
||||
cat > .git/info/sparse-checkout <<'EOF'
|
||||
source/crud/tests/
|
||||
source/unified-test-format/
|
||||
EOF
|
||||
|
||||
echo "fetching $SPEC_COMMIT ..."
|
||||
git fetch -q --depth 1 origin "$SPEC_COMMIT"
|
||||
git checkout -q FETCH_HEAD
|
||||
|
||||
n=$(find source/crud/tests/unified -name '*.json' | wc -l | tr -d ' ')
|
||||
echo "specs at $SPEC_COMMIT ($n crud/unified json files)"
|
||||
743
tests/spec/run.js
Normal file
743
tests/spec/run.js
Normal file
@@ -0,0 +1,743 @@
|
||||
'use strict';
|
||||
// MongoDB unified-test-format runner, pointed at MultiforaDB.
|
||||
//
|
||||
// PLAN D2 makes the official spec suites the gate for command semantics, and
|
||||
// PLAN D7.6 asks for the harness to exist at M0 with a red baseline -- so this
|
||||
// is deliberately a *measuring* tool, not a passing one. What it must never do
|
||||
// is report a pass it did not earn: a compatibility scorecard that flatters
|
||||
// the engine is worse than no scorecard. Everything unimplemented is reported
|
||||
// as SKIP with a reason, never as PASS.
|
||||
//
|
||||
// bash tests/spec/fetch.sh # pinned suites
|
||||
// node tests/spec/run.js # spawns its own server
|
||||
// node tests/spec/run.js --url mongodb://127.0.0.1:27020 # external server
|
||||
// node tests/spec/run.js --file find.json --verbose
|
||||
// node tests/spec/run.js --scorecard # rewrite tests/spec/scorecard.txt
|
||||
//
|
||||
// Matching follows the pseudo-code in the spec's "Evaluating Matches"
|
||||
// section verbatim, including the two rules that are easy to get wrong and
|
||||
// that decide whether a result is a real pass: extra keys are tolerated only
|
||||
// in a *root* document, and numeric types compare flexibly.
|
||||
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const { spawn } = require('child_process');
|
||||
|
||||
const DRIVER = path.join(__dirname, '..', 'e2e', 'node_modules', 'mongodb');
|
||||
const { MongoClient } = require(DRIVER);
|
||||
const { EJSON, Long, Int32, Double, Decimal128, ObjectId, Binary, Timestamp } = require(path.join(DRIVER, 'lib', 'bson.js'));
|
||||
|
||||
const SUITE_DIR = path.join(__dirname, 'specifications', 'source', 'crud', 'tests', 'unified');
|
||||
const SCORECARD = path.join(__dirname, 'scorecard.txt');
|
||||
const REPO = path.join(__dirname, '..', '..');
|
||||
|
||||
// The runner implements schema 1.0-1.9 features that CRUD tests actually use.
|
||||
// A file declaring more than this is skipped whole rather than half-run.
|
||||
const MAX_SCHEMA = [1, 24];
|
||||
|
||||
const argv = process.argv.slice(2);
|
||||
function opt(name, dflt) {
|
||||
const i = argv.indexOf('--' + name);
|
||||
if (i < 0) return dflt;
|
||||
const v = argv[i + 1];
|
||||
return v === undefined || v.startsWith('--') ? true : v;
|
||||
}
|
||||
const VERBOSE = !!opt('verbose', false);
|
||||
const ONLY_FILE = opt('file', null);
|
||||
const WRITE_SCORECARD = !!opt('scorecard', false);
|
||||
const EXTERNAL_URL = opt('url', null);
|
||||
const PORT = parseInt(opt('port', '27222'), 10);
|
||||
const BIN = process.env.MFDB_BIN || path.join(REPO, 'zig-out', 'bin', 'multiforadb');
|
||||
const DB_PATH = process.env.MFDB_SPEC_DB || path.join(REPO, '.zig-cache', 'spec.log');
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Server lifecycle (mirrors tests/e2e/e2e6.js: poll with a real ping, and
|
||||
// treat an exited child as a hard failure so we cannot silently measure a
|
||||
// stale server on the same port).
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
let server = null;
|
||||
let serverDead = false;
|
||||
let serverExit = null;
|
||||
let serverOut = [];
|
||||
// Persisted, not just buffered in memory: when the server dies mid-run its
|
||||
// panic is the only thing that explains the hundreds of ECONNREFUSED failures
|
||||
// that follow, and an in-memory buffer is lost if the runner itself is killed.
|
||||
const SERVER_LOG = path.join(REPO, '.zig-cache', 'spec-server.out');
|
||||
|
||||
function sleep(ms) { return new Promise((r) => setTimeout(r, ms)); }
|
||||
|
||||
// A single case must not be able to stall the baseline run. A timeout is
|
||||
// reported as a FAIL, not a SKIP: "the engine never answered" is a result, and
|
||||
// hiding it in the skip column would overstate compatibility.
|
||||
//
|
||||
// Per-operation, enforced by the driver (CSOT `timeoutMS`) so the promise
|
||||
// settles instead of being abandoned. The budget exists because some commands
|
||||
// produce no reply at all rather than an error, and a driver would otherwise
|
||||
// wait indefinitely.
|
||||
//
|
||||
// 10 s, not the 3 s that seems ample for an in-process engine on loopback
|
||||
// where a clean suite file runs in well under a second. A tight budget makes
|
||||
// the scorecard a function of machine load rather than of the engine: with a
|
||||
// concurrent `zig build test` (which compiles and runs multi-threaded tests),
|
||||
// a 3 s budget turned 166 perfectly good cases into timeout FAILs, and the
|
||||
// same files passed alone. A generous budget costs wall-clock only on cases
|
||||
// that genuinely hang, which are few. Don't tighten it to speed up a run.
|
||||
const OP_TIMEOUT_MS = parseInt(opt('op-timeout-ms', '10000'), 10);
|
||||
// Backstop only, for a hang the driver's own timeout cannot see. It should
|
||||
// essentially never fire -- when it does, the case is abandoned, and abandoned
|
||||
// work is what corrupted two earlier baselines, so it is deliberately far
|
||||
// above OP_TIMEOUT_MS rather than close to it.
|
||||
const CASE_TIMEOUT_MS = parseInt(opt('case-timeout-ms', '60000'), 10);
|
||||
|
||||
// Live handle census. Two baselines were wrecked by the runner accumulating
|
||||
// something across a single process's 175 files, and a count taken only at the
|
||||
// end could not say *when* it started. Printed per file so the transition is
|
||||
// visible in the log.
|
||||
function resourceTag() {
|
||||
if (!process.getActiveResourcesInfo) return '';
|
||||
const counts = {};
|
||||
for (const r of process.getActiveResourcesInfo()) counts[r] = (counts[r] || 0) + 1;
|
||||
const interesting = Object.entries(counts)
|
||||
.filter(([k]) => k !== 'Immediate' && k !== 'TTYWrap' && k !== 'ProcessWrap' && k !== 'PipeWrap')
|
||||
.map(([k, v]) => `${k}=${v}`)
|
||||
.join(' ');
|
||||
return interesting ? ` [${interesting}]` : '';
|
||||
}
|
||||
|
||||
// Server round-trip time, so "the engine is getting slower as the run goes on"
|
||||
// is a number in the log rather than a guess. Also counts databases, since
|
||||
// state accumulating on the server is the obvious candidate.
|
||||
async function pingTag() {
|
||||
try {
|
||||
const t0 = process.hrtime.bigint();
|
||||
await harness.db('admin').command({ ping: 1 });
|
||||
const ms = Number(process.hrtime.bigint() - t0) / 1e6;
|
||||
let ndbs = '?';
|
||||
try {
|
||||
const r = await harness.db('admin').command({ listDatabases: 1, nameOnly: true });
|
||||
ndbs = String((r.databases || []).length);
|
||||
} catch (e) {
|
||||
ndbs = `ERR(${e.name}: ${String(e.message).slice(0, 60)})`;
|
||||
}
|
||||
return `ping=${ms.toFixed(1)}ms dbs=${ndbs}`;
|
||||
} catch (e) {
|
||||
return `ping=FAILED(${e.name})`;
|
||||
}
|
||||
}
|
||||
|
||||
async function withTimeout(fn, ms, msg) {
|
||||
let timer;
|
||||
const guard = new Promise((_, reject) => { timer = setTimeout(() => reject(new Error(msg)), ms); });
|
||||
try { return await Promise.race([fn(), guard]); } finally { clearTimeout(timer); }
|
||||
}
|
||||
|
||||
async function startServer() {
|
||||
try { fs.unlinkSync(DB_PATH); } catch (e) { /* first run */ }
|
||||
if (!fs.existsSync(BIN)) throw new Error(`${BIN} missing - run \`zig build\` first`);
|
||||
server = spawn(BIN, ['--port', String(PORT), '--db', DB_PATH, '--ttl-sweep-secs', '0'], { stdio: ['ignore', 'pipe', 'pipe'] });
|
||||
try { fs.unlinkSync(SERVER_LOG); } catch (e) { /* first run */ }
|
||||
const sink = fs.createWriteStream(SERVER_LOG, { flags: 'a' });
|
||||
const grab = (b) => { sink.write(b); serverOut.push(b.toString()); if (serverOut.length > 400) serverOut.shift(); };
|
||||
server.stdout.on('data', grab);
|
||||
server.stderr.on('data', grab);
|
||||
server.on('exit', (code, signal) => { serverDead = true; serverExit = signal ? `signal ${signal}` : `exit code ${code}`; });
|
||||
|
||||
const url = `mongodb://127.0.0.1:${PORT}`;
|
||||
const deadline = Date.now() + 20000;
|
||||
while (Date.now() < deadline) {
|
||||
if (serverDead) throw new Error('server exited during startup:\n' + serverOut.join(''));
|
||||
try {
|
||||
const c = new MongoClient(url, { serverSelectionTimeoutMS: 500 });
|
||||
await c.connect();
|
||||
await c.db('admin').command({ ping: 1 });
|
||||
await c.close();
|
||||
return url;
|
||||
} catch (e) { await sleep(100); }
|
||||
}
|
||||
throw new Error('server did not become ready:\n' + serverOut.join(''));
|
||||
}
|
||||
|
||||
function stopServer() {
|
||||
if (server && !serverDead) server.kill('SIGKILL');
|
||||
}
|
||||
process.on('exit', stopServer);
|
||||
process.on('SIGINT', () => { stopServer(); process.exit(130); });
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Matching -- the spec's Evaluating Matches algorithm.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
class MatchError extends Error {}
|
||||
|
||||
function fail(pathStr, msg) { throw new MatchError(`${pathStr || '<root>'}: ${msg}`); }
|
||||
|
||||
function isPlainDoc(v) {
|
||||
if (v === null || typeof v !== 'object' || Array.isArray(v)) return false;
|
||||
// BSON scalars are objects too; they are values, not documents.
|
||||
return !(v instanceof ObjectId || v instanceof Binary || v instanceof Long ||
|
||||
v instanceof Int32 || v instanceof Double || v instanceof Decimal128 ||
|
||||
v instanceof Timestamp || v instanceof Date || v instanceof RegExp);
|
||||
}
|
||||
|
||||
function specialKey(doc) {
|
||||
if (!isPlainDoc(doc)) return null;
|
||||
const ks = Object.keys(doc);
|
||||
return ks.length === 1 && ks[0].startsWith('$$') ? ks[0] : null;
|
||||
}
|
||||
|
||||
function numeric(v) {
|
||||
if (typeof v === 'number') return v;
|
||||
if (v instanceof Int32 || v instanceof Double) return v.valueOf();
|
||||
if (v instanceof Long) return Number(v.toString());
|
||||
return null;
|
||||
}
|
||||
|
||||
// The spec's type aliases, restricted to what the CRUD suites use.
|
||||
function matchesType(actual, alias) {
|
||||
switch (alias) {
|
||||
case 'double': return typeof actual === 'number' || actual instanceof Double;
|
||||
case 'int': return actual instanceof Int32 || (typeof actual === 'number' && Number.isInteger(actual));
|
||||
case 'long': return actual instanceof Long || (typeof actual === 'number' && Number.isInteger(actual));
|
||||
case 'string': return typeof actual === 'string';
|
||||
case 'object': return isPlainDoc(actual);
|
||||
case 'array': return Array.isArray(actual);
|
||||
case 'binData': return actual instanceof Binary;
|
||||
case 'objectId': return actual instanceof ObjectId;
|
||||
case 'bool': return typeof actual === 'boolean';
|
||||
case 'date': return actual instanceof Date;
|
||||
case 'null': return actual === null;
|
||||
case 'regex': return actual instanceof RegExp;
|
||||
case 'timestamp': return actual instanceof Timestamp;
|
||||
case 'decimal': return actual instanceof Decimal128;
|
||||
case 'number': return numeric(actual) !== null || actual instanceof Decimal128;
|
||||
default: return null; // unknown alias -> caller reports unsupported
|
||||
}
|
||||
}
|
||||
|
||||
function scalarEqual(expected, actual) {
|
||||
const en = numeric(expected), an = numeric(actual);
|
||||
if (en !== null && an !== null) return en === an; // flexible numerics
|
||||
if (expected === null) return actual === null;
|
||||
if (expected instanceof Date) return actual instanceof Date && +expected === +actual;
|
||||
if (expected instanceof ObjectId) return actual instanceof ObjectId && expected.equals(actual);
|
||||
if (expected instanceof Binary) return actual instanceof Binary && Buffer.compare(expected.buffer, actual.buffer) === 0 && expected.sub_type === actual.sub_type;
|
||||
if (expected instanceof Decimal128) return actual instanceof Decimal128 && expected.toString() === actual.toString();
|
||||
if (expected instanceof Timestamp) return actual instanceof Timestamp && expected.equals(actual);
|
||||
if (expected instanceof RegExp) return actual instanceof RegExp && String(expected) === String(actual);
|
||||
if (typeof expected !== typeof actual) return false;
|
||||
return expected === actual;
|
||||
}
|
||||
|
||||
// `root` implements the one rule that separates a real pass from a lenient
|
||||
// one: only a root document may carry keys the expectation does not mention.
|
||||
function match(expected, actual, entities, pathStr = '', root = true) {
|
||||
const sk = specialKey(expected);
|
||||
if (sk) return special(sk, expected[sk], actual, entities, pathStr, true);
|
||||
|
||||
if (isPlainDoc(expected)) {
|
||||
if (!isPlainDoc(actual)) fail(pathStr, `expected a document, got ${describe(actual)}`);
|
||||
for (const [k, v] of Object.entries(expected)) {
|
||||
const kp = pathStr ? `${pathStr}.${k}` : k;
|
||||
const vsk = specialKey(v);
|
||||
if (vsk) {
|
||||
const consumed = special(vsk, v[vsk], actual[k], entities, kp, Object.prototype.hasOwnProperty.call(actual, k));
|
||||
if (!consumed) continue;
|
||||
continue;
|
||||
}
|
||||
if (!Object.prototype.hasOwnProperty.call(actual, k)) fail(kp, 'missing from actual');
|
||||
match(v, actual[k], entities, kp, false);
|
||||
}
|
||||
if (!root) {
|
||||
const extra = Object.keys(actual).filter((k) => !Object.prototype.hasOwnProperty.call(expected, k));
|
||||
if (extra.length) fail(pathStr, `unexpected extra keys ${JSON.stringify(extra)}`);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
if (Array.isArray(expected)) {
|
||||
if (!Array.isArray(actual)) fail(pathStr, `expected an array, got ${describe(actual)}`);
|
||||
if (expected.length !== actual.length) fail(pathStr, `expected ${expected.length} elements, got ${actual.length}`);
|
||||
// Array elements are not root documents: `expectResult: [ {...} ]` from
|
||||
// find must match its documents exactly, extra fields included.
|
||||
expected.forEach((e, i) => match(e, actual[i], entities, `${pathStr}[${i}]`, false));
|
||||
return;
|
||||
}
|
||||
|
||||
if (!scalarEqual(expected, actual)) fail(pathStr, `expected ${describe(expected)}, got ${describe(actual)}`);
|
||||
}
|
||||
|
||||
// Returns whether the actual value still has to be matched by the caller.
|
||||
function special(op, arg, actual, entities, pathStr, present) {
|
||||
switch (op) {
|
||||
case '$$exists':
|
||||
if (arg && !present) fail(pathStr, 'expected the key to exist');
|
||||
if (!arg && present) fail(pathStr, 'expected the key to be absent');
|
||||
return false;
|
||||
case '$$type': {
|
||||
const aliases = Array.isArray(arg) ? arg : [arg];
|
||||
const results = aliases.map((a) => matchesType(actual, a));
|
||||
if (results.some((r) => r === null)) throw new Unsupported(`$$type alias ${JSON.stringify(arg)}`);
|
||||
if (!results.some((r) => r === true)) fail(pathStr, `expected type ${JSON.stringify(arg)}, got ${describe(actual)}`);
|
||||
return false;
|
||||
}
|
||||
case '$$unsetOrMatches':
|
||||
if (!present || actual === undefined) return false;
|
||||
match(arg, actual, entities, pathStr, false);
|
||||
return false;
|
||||
case '$$matchesEntity': {
|
||||
if (!(arg in entities.map)) fail(pathStr, `entity ${arg} not found`);
|
||||
match(entities.map[arg], actual, entities, pathStr, false);
|
||||
return false;
|
||||
}
|
||||
case '$$matchesHexBytes':
|
||||
if (!(actual instanceof Binary)) fail(pathStr, 'expected binary data');
|
||||
if (actual.buffer.toString('hex') !== String(arg).toLowerCase()) fail(pathStr, 'hex bytes differ');
|
||||
return false;
|
||||
default:
|
||||
throw new Unsupported(`match operator ${op}`);
|
||||
}
|
||||
}
|
||||
|
||||
function describe(v) {
|
||||
if (v === undefined) return 'undefined';
|
||||
if (v === null) return 'null';
|
||||
try { return EJSON.stringify(v, { relaxed: true }); } catch (e) { return String(v); }
|
||||
}
|
||||
|
||||
class Unsupported extends Error {}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Operations
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
// Argument keys the spec passes positionally rather than as driver options.
|
||||
const POSITIONAL = new Set(['filter', 'document', 'documents', 'update', 'replacement', 'pipeline', 'fieldName', 'models', 'requests', 'keys', 'name', 'indexes', 'command', 'session', 'entity', 'to']);
|
||||
|
||||
function options(args, drop = []) {
|
||||
const o = {};
|
||||
for (const [k, v] of Object.entries(args || {})) {
|
||||
if (POSITIONAL.has(k) || drop.includes(k)) continue;
|
||||
o[k] = v;
|
||||
}
|
||||
return Object.keys(o).length ? o : undefined;
|
||||
}
|
||||
|
||||
function requireNoSession(args) {
|
||||
if (args && args.session) throw new Unsupported('explicit sessions (M4)');
|
||||
}
|
||||
|
||||
async function runOperation(op, entities) {
|
||||
const args = op.arguments || {};
|
||||
requireNoSession(args);
|
||||
const target = entities.map[op.object];
|
||||
if (op.object === 'testRunner') throw new Unsupported(`testRunner operation ${op.name}`);
|
||||
if (target === undefined) throw new Unsupported(`entity ${op.object}`);
|
||||
|
||||
switch (op.name) {
|
||||
// -- read ----------------------------------------------------------
|
||||
case 'find': return await target.find(args.filter || {}, options(args)).toArray();
|
||||
case 'findOne': return await target.findOne(args.filter || {}, options(args));
|
||||
case 'aggregate': return await target.aggregate(args.pipeline, options(args)).toArray();
|
||||
case 'countDocuments': return await target.countDocuments(args.filter || {}, options(args));
|
||||
case 'estimatedDocumentCount': return await target.estimatedDocumentCount(options(args));
|
||||
case 'distinct': return await target.distinct(args.fieldName, args.filter || {}, options(args));
|
||||
case 'listIndexes': return await target.listIndexes(options(args)).toArray();
|
||||
case 'listIndexNames': return await target.indexes(options(args)).then((ix) => ix.map((i) => i.name));
|
||||
case 'listCollections': return await target.listCollections(args.filter || {}, options(args)).toArray();
|
||||
case 'listCollectionNames': return await target.listCollections(args.filter || {}, options(args)).toArray().then((cs) => cs.map((c) => c.name));
|
||||
case 'runCommand': return await target.command(args.command, options(args, ['commandName']));
|
||||
|
||||
// -- write ---------------------------------------------------------
|
||||
case 'insertOne': return plain(await target.insertOne(args.document, options(args)));
|
||||
case 'insertMany': return plain(await target.insertMany(args.documents, options(args)));
|
||||
case 'updateOne': return plain(await target.updateOne(args.filter, args.update, options(args)));
|
||||
case 'updateMany': return plain(await target.updateMany(args.filter, args.update, options(args)));
|
||||
case 'replaceOne': return plain(await target.replaceOne(args.filter, args.replacement, options(args)));
|
||||
case 'deleteOne': return plain(await target.deleteOne(args.filter, options(args)));
|
||||
case 'deleteMany': return plain(await target.deleteMany(args.filter, options(args)));
|
||||
case 'bulkWrite': return plain(await target.bulkWrite(args.requests, options(args)));
|
||||
case 'findOneAndUpdate': return unwrapFam(await target.findOneAndUpdate(args.filter, args.update, options(args)));
|
||||
case 'findOneAndReplace': return unwrapFam(await target.findOneAndReplace(args.filter, args.replacement, options(args)));
|
||||
case 'findOneAndDelete': return unwrapFam(await target.findOneAndDelete(args.filter, options(args)));
|
||||
|
||||
// -- collection / index management ----------------------------------
|
||||
case 'createIndex': return await target.createIndex(args.keys, options(args));
|
||||
case 'dropIndex': return await target.dropIndex(args.name, options(args));
|
||||
case 'createCollection': return void (await target.createCollection(args.collection, options(args, ['collection'])));
|
||||
case 'dropCollection': return void (await target.dropCollection(args.collection, options(args, ['collection'])));
|
||||
case 'assertCollectionExists': case 'assertCollectionNotExists':
|
||||
throw new Unsupported(`testRunner operation ${op.name}`);
|
||||
default:
|
||||
throw new Unsupported(`operation ${op.name}`);
|
||||
}
|
||||
}
|
||||
|
||||
// Driver result objects carry class instances and `acknowledged`; the spec
|
||||
// expects plain documents. Turning them into plain objects keeps the
|
||||
// root-document rule meaningful (extra keys tolerated only at the root).
|
||||
function plain(r) {
|
||||
if (r === null || typeof r !== 'object') return r;
|
||||
const o = {};
|
||||
for (const k of Object.keys(r)) {
|
||||
if (k === 'acknowledged') continue;
|
||||
o[k] = r[k];
|
||||
}
|
||||
if (typeof r.getUpsertedIds === 'function') {
|
||||
// BulkWriteResult exposes counts through getters, not own keys.
|
||||
for (const k of ['insertedCount', 'matchedCount', 'modifiedCount', 'deletedCount', 'upsertedCount', 'insertedIds', 'upsertedIds']) {
|
||||
if (r[k] !== undefined) o[k] = r[k];
|
||||
}
|
||||
}
|
||||
return o;
|
||||
}
|
||||
|
||||
// Driver 5+ returns the document itself; 4.x wrapped it in `{value}`. e2e.js
|
||||
// tolerates both the same way (`fam.value ?? fam`).
|
||||
function unwrapFam(r) {
|
||||
if (r && typeof r === 'object' && 'value' in r && 'ok' in r) return r.value;
|
||||
return r;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// runOnRequirements
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function cmpVersion(a, b) {
|
||||
const pa = String(a).split('.').map(Number), pb = String(b).split('.').map(Number);
|
||||
for (let i = 0; i < Math.max(pa.length, pb.length); i++) {
|
||||
const x = pa[i] || 0, y = pb[i] || 0;
|
||||
if (x !== y) return x < y ? -1 : 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
function unmetRequirement(reqs, server) {
|
||||
if (!reqs) return null;
|
||||
const reasons = [];
|
||||
for (const r of reqs) {
|
||||
const why = [];
|
||||
if (r.minServerVersion && cmpVersion(server.version, r.minServerVersion) < 0) why.push(`needs server >= ${r.minServerVersion}`);
|
||||
if (r.maxServerVersion && cmpVersion(server.version, r.maxServerVersion) > 0) why.push(`needs server <= ${r.maxServerVersion}`);
|
||||
if (r.topologies && !r.topologies.includes('single')) why.push(`needs topology ${r.topologies.join('/')}`);
|
||||
if (r.auth === true) why.push('needs auth (M7)');
|
||||
if (r.csfle) why.push('needs csfle');
|
||||
if (r.serverless === 'require') why.push('needs serverless');
|
||||
if (r.serverParameters) why.push('needs server parameters');
|
||||
if (why.length === 0) return null; // this alternative is satisfied
|
||||
reasons.push(why.join(', '));
|
||||
}
|
||||
return reasons[0] || 'unmet requirement';
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// One file
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
// One client for the whole run, used for seeding and outcome checks.
|
||||
//
|
||||
// It used to be one client per seed and per outcome check. That, plus per-test
|
||||
// entity clients that leaked whenever a case timed out before buildEntities
|
||||
// returned, meant hundreds of live MongoClients accumulated -- each with its
|
||||
// own pool and heartbeat timer. Node's event loop starved, the per-case timer
|
||||
// then fired before operations could complete, and every case from that point
|
||||
// on "failed" with a timeout. A ping from a separate process still answered,
|
||||
// which made it look convincingly like a server-side wedge. It was the runner.
|
||||
//
|
||||
// The lesson worth keeping: a harness whose own failures look like engine
|
||||
// failures will silently invent a compatibility gap. Hence also the assertion
|
||||
// below that clients never accumulate.
|
||||
let harness = null;
|
||||
|
||||
async function seedInitialData(initialData) {
|
||||
if (!initialData) return;
|
||||
for (const spec of initialData) {
|
||||
const coll = harness.db(spec.databaseName).collection(spec.collectionName);
|
||||
await coll.drop().catch(() => { });
|
||||
if (spec.documents && spec.documents.length) await coll.insertMany(spec.documents);
|
||||
else await harness.db(spec.databaseName).createCollection(spec.collectionName).catch(() => { });
|
||||
}
|
||||
}
|
||||
|
||||
// `clients` is supplied by the caller so that entities created before a
|
||||
// failure are still closed: returning them only on success is what leaked.
|
||||
async function buildEntities(url, createEntities, clients) {
|
||||
const map = {};
|
||||
for (const spec of createEntities || []) {
|
||||
const [kind, def] = Object.entries(spec)[0];
|
||||
switch (kind) {
|
||||
case 'client': {
|
||||
if (def.useMultipleMongoses === true) { /* single topology: harmless */ }
|
||||
// `timeoutMS` (CSOT) makes each operation reject on its own
|
||||
// rather than being abandoned by an outer race. That matters
|
||||
// more than it looks: an abandoned operation keeps running,
|
||||
// keeps its client alive, and -- since it may still be inside
|
||||
// buildEntities -- can create further clients *after* cleanup
|
||||
// has already run. That is what leaked, and what turned into
|
||||
// 190 phantom timeout FAILs.
|
||||
const c = new MongoClient(url, Object.assign({
|
||||
serverSelectionTimeoutMS: 2000,
|
||||
connectTimeoutMS: 2000,
|
||||
timeoutMS: OP_TIMEOUT_MS,
|
||||
}, def.uriOptions || {}));
|
||||
// Registered before connect, so a client whose connect throws
|
||||
// or is abandoned is still closed by the caller.
|
||||
clients.push(c);
|
||||
await c.connect();
|
||||
if (clients.abandoned) {
|
||||
// Cleanup already ran; this client would otherwise linger.
|
||||
await c.close().catch(() => { });
|
||||
throw new Error('case abandoned');
|
||||
}
|
||||
map[def.id] = c;
|
||||
break;
|
||||
}
|
||||
case 'database': map[def.id] = map[def.client].db(def.databaseName); break;
|
||||
case 'collection': map[def.id] = map[def.database].collection(def.collectionName); break;
|
||||
case 'session': throw new Unsupported('session entities (M4)');
|
||||
case 'bucket': throw new Unsupported('gridfs bucket entities');
|
||||
case 'clientEncryption': throw new Unsupported('client-side encryption');
|
||||
default: throw new Unsupported(`entity type ${kind}`);
|
||||
}
|
||||
}
|
||||
return { map, clients };
|
||||
}
|
||||
|
||||
async function verifyOutcome(outcome, entities) {
|
||||
if (!outcome) return;
|
||||
for (const spec of outcome) {
|
||||
const actual = await harness.db(spec.databaseName).collection(spec.collectionName)
|
||||
.find({}, { sort: { _id: 1 } }).toArray();
|
||||
match(spec.documents, actual, entities, `outcome ${spec.databaseName}.${spec.collectionName}`, false);
|
||||
}
|
||||
}
|
||||
|
||||
async function runFile(file, url, server) {
|
||||
const text = fs.readFileSync(path.join(SUITE_DIR, file), 'utf8');
|
||||
const doc = EJSON.parse(text, { relaxed: false });
|
||||
const out = { file, pass: 0, fail: 0, skip: 0, cases: [] };
|
||||
|
||||
const schema = String(doc.schemaVersion || '1.0').split('.').map(Number);
|
||||
if (schema[0] > MAX_SCHEMA[0] || (schema[0] === MAX_SCHEMA[0] && (schema[1] || 0) > MAX_SCHEMA[1])) {
|
||||
out.skip = (doc.tests || []).length;
|
||||
out.cases.push({ name: '*', status: 'SKIP', reason: `schemaVersion ${doc.schemaVersion} > ${MAX_SCHEMA.join('.')}` });
|
||||
return out;
|
||||
}
|
||||
const fileUnmet = unmetRequirement(doc.runOnRequirements, server);
|
||||
if (fileUnmet) {
|
||||
out.skip = (doc.tests || []).length;
|
||||
out.cases.push({ name: '*', status: 'SKIP', reason: fileUnmet });
|
||||
return out;
|
||||
}
|
||||
|
||||
for (const test of doc.tests || []) {
|
||||
const name = test.description;
|
||||
if (test.skipReason) { out.skip++; out.cases.push({ name, status: 'SKIP', reason: 'upstream skipReason: ' + test.skipReason }); continue; }
|
||||
const unmet = unmetRequirement(test.runOnRequirements, server);
|
||||
if (unmet) { out.skip++; out.cases.push({ name, status: 'SKIP', reason: unmet }); continue; }
|
||||
|
||||
// Owned out here, not by buildEntities, so a case that dies partway
|
||||
// still has every client it managed to open closed below.
|
||||
const clients = [];
|
||||
try {
|
||||
await withTimeout(async () => {
|
||||
await seedInitialData(doc.initialData);
|
||||
const entities = await buildEntities(url, doc.createEntities, clients);
|
||||
for (const op of test.operations) await runOne(op, entities);
|
||||
await verifyOutcome(test.outcome, entities);
|
||||
}, CASE_TIMEOUT_MS, `case exceeded ${CASE_TIMEOUT_MS} ms`);
|
||||
out.pass++;
|
||||
out.cases.push({ name, status: 'PASS' });
|
||||
} catch (e) {
|
||||
if (e instanceof Unsupported) { out.skip++; out.cases.push({ name, status: 'SKIP', reason: 'runner: ' + e.message }); }
|
||||
else { out.fail++; out.cases.push({ name, status: 'FAIL', reason: (e instanceof MatchError ? '' : (e.constructor.name + ': ')) + e.message.split('\n')[0].slice(0, 220) }); }
|
||||
} finally {
|
||||
// Unconditional, and the flag matters: if the case was abandoned,
|
||||
// its continuation may still be running and about to open another
|
||||
// client, which buildEntities closes itself on seeing this.
|
||||
clients.abandoned = true;
|
||||
for (const c of clients) await c.close().catch(() => { });
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
async function runOne(op, entities) {
|
||||
if (op.name === 'failPoint' || op.name === 'targetedFailPoint') throw new Unsupported('failPoint');
|
||||
let result, err = null;
|
||||
try {
|
||||
result = await runOperation(op, entities);
|
||||
} catch (e) {
|
||||
if (e instanceof Unsupported) throw e;
|
||||
err = e;
|
||||
}
|
||||
|
||||
if (op.expectError) {
|
||||
if (!err) fail(op.name, 'expected an error, the operation succeeded');
|
||||
checkError(op.expectError, err, entities, op.name);
|
||||
return;
|
||||
}
|
||||
if (err) throw err;
|
||||
if (op.ignoreResultAndError) return;
|
||||
if ('expectResult' in op) match(op.expectResult, result, entities, op.name, true);
|
||||
if (op.saveResultAsEntity) entities.map[op.saveResultAsEntity] = result;
|
||||
}
|
||||
|
||||
function checkError(exp, err, entities, where) {
|
||||
if (exp.isError === false) fail(where, 'expected no error');
|
||||
if (exp.isClientError === true && err.code !== undefined && err.code !== null) {
|
||||
fail(where, `expected a client-side error, got server code ${err.code}`);
|
||||
}
|
||||
if (exp.errorContains) {
|
||||
const hay = String(err.message || '').toLowerCase();
|
||||
if (!hay.includes(String(exp.errorContains).toLowerCase())) fail(where, `error message ${JSON.stringify(String(err.message).slice(0, 120))} does not contain ${JSON.stringify(exp.errorContains)}`);
|
||||
}
|
||||
if (exp.errorCode !== undefined) {
|
||||
const actual = numeric(err.code);
|
||||
if (actual !== numeric(exp.errorCode)) fail(where, `expected error code ${exp.errorCode}, got ${err.code}`);
|
||||
}
|
||||
if (exp.errorCodeName !== undefined) {
|
||||
const actual = err.codeName || (err.result && err.result.codeName);
|
||||
if (actual !== exp.errorCodeName) fail(where, `expected codeName ${exp.errorCodeName}, got ${actual}`);
|
||||
}
|
||||
if (exp.errorLabelsContain) {
|
||||
for (const l of exp.errorLabelsContain) if (!(err.errorLabels || []).includes(l)) fail(where, `expected error label ${l}`);
|
||||
}
|
||||
if (exp.errorLabelsOmit) {
|
||||
for (const l of exp.errorLabelsOmit) if ((err.errorLabels || []).includes(l)) fail(where, `expected no error label ${l}`);
|
||||
}
|
||||
if (exp.errorResponse) {
|
||||
const resp = err.result || err.errorResponse || err;
|
||||
match(exp.errorResponse, resp, entities, where + '.errorResponse', true);
|
||||
}
|
||||
if (exp.writeErrors || exp.writeConcernErrors) throw new Unsupported('clientBulkWrite error assertions');
|
||||
if ('expectResult' in exp) {
|
||||
const r = err.result || (err.writeErrors ? plain(err) : undefined);
|
||||
if (r === undefined) throw new Unsupported('expectResult on an error the driver does not expose');
|
||||
match(exp.expectResult, r, entities, where + '.expectResult', true);
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// main
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
(async () => {
|
||||
if (!fs.existsSync(SUITE_DIR)) {
|
||||
console.error(`missing ${SUITE_DIR}\nrun: bash tests/spec/fetch.sh`);
|
||||
process.exit(2);
|
||||
}
|
||||
let files = fs.readdirSync(SUITE_DIR).filter((f) => f.endsWith('.json'))
|
||||
.filter((f) => !ONLY_FILE || f === ONLY_FILE || f === ONLY_FILE + '.json')
|
||||
.sort();
|
||||
// --skip/--limit exist for bisecting a run whose failures depend on
|
||||
// position rather than content, which has happened more than once.
|
||||
const skip = parseInt(opt('skip', '0'), 10);
|
||||
if (skip > 0) files = files.slice(skip);
|
||||
const limit = parseInt(opt('limit', '0'), 10);
|
||||
if (limit > 0) files = files.slice(0, limit);
|
||||
if (!files.length) { console.error('no matching suite files'); process.exit(2); }
|
||||
|
||||
const url = EXTERNAL_URL || await startServer();
|
||||
|
||||
harness = new MongoClient(url, { serverSelectionTimeoutMS: 3000, connectTimeoutMS: 3000, socketTimeoutMS: 8000 });
|
||||
await harness.connect();
|
||||
const bi = await harness.db('admin').command({ buildInfo: 1 }).catch(() => ({ version: '0.0.0' }));
|
||||
const hello = await harness.db('admin').command({ hello: 1 }).catch(() => ({}));
|
||||
const server = { version: bi.version || '0.0.0', maxWireVersion: hello.maxWireVersion };
|
||||
|
||||
const results = [];
|
||||
let died = null;
|
||||
for (const f of files) {
|
||||
// A dead server turns every remaining case into ECONNREFUSED, which
|
||||
// would land in the scorecard as ~700 engine failures and bury the one
|
||||
// fact that matters: it crashed, and where. Stop at the crash instead.
|
||||
if (serverDead && !EXTERNAL_URL) {
|
||||
died = { after: results.length ? results[results.length - 1].file : '<startup>', next: f };
|
||||
break;
|
||||
}
|
||||
let r;
|
||||
try {
|
||||
r = await runFile(f, url, server);
|
||||
} catch (e) {
|
||||
r = { file: f, pass: 0, fail: 0, skip: 0, cases: [{ name: '*', status: 'ERROR', reason: e.message.split('\n')[0].slice(0, 200) }], errored: true };
|
||||
}
|
||||
results.push(r);
|
||||
const tag = r.errored ? 'ERROR' : `${r.pass} pass, ${r.fail} fail, ${r.skip} skip`;
|
||||
console.log(`${f.padEnd(48)} ${tag}${resourceTag()} ${await pingTag()}`);
|
||||
if (VERBOSE) for (const c of r.cases) if (c.status !== 'PASS') console.log(` ${c.status.padEnd(5)} ${c.name}${c.reason ? ' -- ' + c.reason : ''}`);
|
||||
}
|
||||
|
||||
const tot = results.reduce((a, r) => ({ pass: a.pass + r.pass, fail: a.fail + r.fail, skip: a.skip + r.skip }), { pass: 0, fail: 0, skip: 0 });
|
||||
const errored = results.filter((r) => r.errored).length;
|
||||
console.log(`\ncrud+aggregate: ${tot.pass} pass, ${tot.fail} fail, ${tot.skip} skip across ${results.length}/${files.length} files (${errored} files errored)`);
|
||||
|
||||
if (died) {
|
||||
console.error(`\n*** THE SERVER DIED (${serverExit}) ***`);
|
||||
console.error(`last file completed: ${died.after}; would have run next: ${died.next}`);
|
||||
console.error(`server output: ${path.relative(REPO, SERVER_LOG)}`);
|
||||
console.error('--- tail ---\n' + serverOut.slice(-40).join(''));
|
||||
console.error('Refusing to write a scorecard from a partial run.');
|
||||
process.exit(3);
|
||||
}
|
||||
|
||||
if (WRITE_SCORECARD) {
|
||||
fs.writeFileSync(SCORECARD, scorecardText(results, tot, errored, server, files.length));
|
||||
console.log(`wrote ${path.relative(REPO, SCORECARD)}`);
|
||||
}
|
||||
// A runner leak once turned into ~77 phantom "engine timeouts" (see the
|
||||
// comment on `harness`). This is the tripwire for that class of bug: if the
|
||||
// harness is the only client left standing, nothing accumulated.
|
||||
const leaked = process.getActiveResourcesInfo
|
||||
? process.getActiveResourcesInfo().filter((r) => r === 'Timeout').length
|
||||
: 0;
|
||||
if (leaked > 8) {
|
||||
console.error(`\nWARNING: ${leaked} timers still active at the end of the run.`);
|
||||
console.error('That is how a client leak looks; treat the timeout FAILs above as suspect.');
|
||||
}
|
||||
await harness.close(true).catch(() => { });
|
||||
|
||||
// The runner's own exit code reports whether it *ran*, not whether the
|
||||
// engine passed -- M0's gate is a recorded baseline, and a red baseline is
|
||||
// the expected state. A non-zero exit here would make it useless as a gate.
|
||||
console.log('SPEC_RUNNER_OK');
|
||||
stopServer();
|
||||
process.exit(0);
|
||||
})().catch((e) => {
|
||||
console.error('spec runner failed:', e);
|
||||
if (serverOut.length) console.error('--- server output ---\n' + serverOut.slice(-30).join(''));
|
||||
stopServer();
|
||||
process.exit(1);
|
||||
});
|
||||
|
||||
function scorecardText(results, tot, errored, server, nfiles) {
|
||||
const L = [];
|
||||
L.push('# MongoDB spec-test scorecard (PLAN D2) -- crud + aggregate, unified format');
|
||||
L.push(`# specs: mongodb/specifications @ 615e0f9 (pinned in tests/spec/fetch.sh)`);
|
||||
L.push(`# driver: mongodb@${require(path.join(DRIVER, 'package.json')).version} (pinned in tests/e2e/package-lock.json)`);
|
||||
L.push(`# server: MultiforaDB reporting version ${server.version}, maxWireVersion ${server.maxWireVersion}`);
|
||||
L.push('# reproduce: bash tests/spec/fetch.sh && node tests/spec/run.js --scorecard');
|
||||
L.push('#');
|
||||
L.push('# What SKIP means here, so the totals are not read as better than they are:');
|
||||
L.push('# - the suite needs a feature whose milestone has not landed (sessions M4,');
|
||||
L.push('# auth M7, failPoints, gridfs) -- counted as skip, never as pass;');
|
||||
L.push('# - or the runner itself does not implement the operation/matcher yet.');
|
||||
L.push('# Deliberately NOT asserted yet: expectEvents (command monitoring). Those');
|
||||
L.push('# assertions are about driver-visible command shape rather than result');
|
||||
L.push('# semantics; ignoring them makes some cases pass that a full runner would');
|
||||
L.push('# fail, so treat `pass` as an upper bound until M1 wires events up.');
|
||||
L.push('');
|
||||
L.push(`total\t${tot.pass} pass\t${tot.fail} fail\t${tot.skip} skip\t${nfiles} files\t${errored} errored`);
|
||||
L.push('');
|
||||
L.push('# per-file: name\tpass\tfail\tskip');
|
||||
for (const r of results) L.push(`${r.file}\t${r.pass}\t${r.fail}\t${r.skip}${r.errored ? '\terrored' : ''}`);
|
||||
L.push('');
|
||||
L.push('# every non-passing case, with its reason');
|
||||
for (const r of results) {
|
||||
for (const c of r.cases) {
|
||||
if (c.status === 'PASS') continue;
|
||||
L.push(`${r.file}\t${c.status}\t${c.name}\t${c.reason || ''}`);
|
||||
}
|
||||
}
|
||||
return L.join('\n') + '\n';
|
||||
}
|
||||
503
tests/spec/scorecard.txt
Normal file
503
tests/spec/scorecard.txt
Normal file
@@ -0,0 +1,503 @@
|
||||
# MongoDB spec-test scorecard (PLAN D2) -- crud + aggregate, unified format
|
||||
# specs: mongodb/specifications @ 615e0f9 (pinned in tests/spec/fetch.sh)
|
||||
# driver: mongodb@7.5.0 (pinned in tests/e2e/package-lock.json)
|
||||
# server: MultiforaDB reporting version 4.4.0, maxWireVersion 8
|
||||
# reproduce: bash tests/spec/fetch.sh && node tests/spec/run.js --scorecard
|
||||
#
|
||||
# What SKIP means here, so the totals are not read as better than they are:
|
||||
# - the suite needs a feature whose milestone has not landed (sessions M4,
|
||||
# auth M7, failPoints, gridfs) -- counted as skip, never as pass;
|
||||
# - or the runner itself does not implement the operation/matcher yet.
|
||||
# Deliberately NOT asserted yet: expectEvents (command monitoring). Those
|
||||
# assertions are about driver-visible command shape rather than result
|
||||
# semantics; ignoring them makes some cases pass that a full runner would
|
||||
# fail, so treat `pass` as an upper bound until M1 wires events up.
|
||||
|
||||
total 131 pass 161 fail 195 skip 175 files 0 errored
|
||||
|
||||
# per-file: name pass fail skip
|
||||
aggregate-allowdiskuse.json 3 0 0
|
||||
aggregate-collation.json 0 1 0
|
||||
aggregate-let.json 0 2 2
|
||||
aggregate-merge-errorResponse.json 0 0 1
|
||||
aggregate-merge.json 0 5 0
|
||||
aggregate-out-readConcern.json 0 0 4
|
||||
aggregate-out.json 0 2 0
|
||||
aggregate-rawdata.json 1 0 1
|
||||
aggregate-write-readPreference.json 0 0 4
|
||||
aggregate.json 5 0 2
|
||||
bulkWrite-arrayFilters.json 0 3 0
|
||||
bulkWrite-collation.json 0 2 0
|
||||
bulkWrite-comment.json 0 2 1
|
||||
bulkWrite-delete-hint-serverError.json 0 0 2
|
||||
bulkWrite-delete-hint.json 2 0 0
|
||||
bulkWrite-deleteMany-hint-unacknowledged.json 0 2 2
|
||||
bulkWrite-deleteMany-let.json 0 1 1
|
||||
bulkWrite-deleteMany-rawdata.json 1 0 1
|
||||
bulkWrite-deleteOne-hint-unacknowledged.json 0 2 2
|
||||
bulkWrite-deleteOne-let.json 0 1 1
|
||||
bulkWrite-deleteOne-rawdata.json 1 0 1
|
||||
bulkWrite-errorResponse.json 0 0 1
|
||||
bulkWrite-insertOne-dots_and_dollars.json 3 1 1
|
||||
bulkWrite-replaceOne-dots_and_dollars.json 1 2 1
|
||||
bulkWrite-replaceOne-hint-unacknowledged.json 0 2 0
|
||||
bulkWrite-replaceOne-let.json 0 1 1
|
||||
bulkWrite-replaceOne-rawdata.json 1 0 1
|
||||
bulkWrite-replaceOne-sort.json 1 0 1
|
||||
bulkWrite-update-hint.json 2 1 0
|
||||
bulkWrite-update-validation.json 3 0 0
|
||||
bulkWrite-updateMany-dots_and_dollars.json 0 0 4
|
||||
bulkWrite-updateMany-hint-unacknowledged.json 0 2 0
|
||||
bulkWrite-updateMany-let.json 0 1 1
|
||||
bulkWrite-updateMany-pipeline.json 0 1 0
|
||||
bulkWrite-updateMany-rawdata.json 0 1 1
|
||||
bulkWrite-updateOne-dots_and_dollars.json 0 0 4
|
||||
bulkWrite-updateOne-hint-unacknowledged.json 0 2 0
|
||||
bulkWrite-updateOne-let.json 0 1 1
|
||||
bulkWrite-updateOne-pipeline.json 0 1 0
|
||||
bulkWrite-updateOne-rawdata.json 0 1 1
|
||||
bulkWrite-updateOne-sort.json 1 0 1
|
||||
bulkWrite.json 5 5 0
|
||||
bypassDocumentValidation.json 6 3 0
|
||||
client-bulkWrite-delete-options.json 0 0 2
|
||||
client-bulkWrite-delete-rawdata.json 0 0 2
|
||||
client-bulkWrite-errorResponse.json 0 0 1
|
||||
client-bulkWrite-errors.json 0 0 9
|
||||
client-bulkWrite-mixed-namespaces.json 0 0 1
|
||||
client-bulkWrite-options.json 0 0 7
|
||||
client-bulkWrite-ordered.json 0 0 3
|
||||
client-bulkWrite-partialResults.json 0 0 9
|
||||
client-bulkWrite-replaceOne-rawdata.json 0 0 2
|
||||
client-bulkWrite-replaceOne-sort.json 0 0 1
|
||||
client-bulkWrite-results.json 0 0 3
|
||||
client-bulkWrite-update-options.json 0 0 4
|
||||
client-bulkWrite-update-pipeline.json 0 0 2
|
||||
client-bulkWrite-update-rawdata.json 0 0 2
|
||||
client-bulkWrite-update-validation.json 0 0 3
|
||||
client-bulkWrite-updateOne-sort.json 0 0 1
|
||||
count-collation.json 1 0 1
|
||||
count-empty.json 2 0 1
|
||||
count-rawdata.json 0 0 2
|
||||
count.json 3 1 3
|
||||
countDocuments-comment.json 2 0 1
|
||||
countDocuments-rawdata.json 1 0 1
|
||||
create-null-ids.json 0 6 1
|
||||
db-aggregate-rawdata.json 0 1 1
|
||||
db-aggregate-write-readPreference.json 0 0 4
|
||||
db-aggregate.json 0 2 0
|
||||
deleteMany-collation.json 0 1 0
|
||||
deleteMany-comment.json 2 0 1
|
||||
deleteMany-hint-serverError.json 0 0 2
|
||||
deleteMany-hint-unacknowledged.json 0 2 2
|
||||
deleteMany-hint.json 2 0 0
|
||||
deleteMany-let.json 0 1 1
|
||||
deleteMany-rawdata.json 1 0 1
|
||||
deleteMany.json 2 0 0
|
||||
deleteOne-collation.json 0 1 0
|
||||
deleteOne-comment.json 2 0 1
|
||||
deleteOne-errorResponse.json 0 0 1
|
||||
deleteOne-hint-serverError.json 0 0 2
|
||||
deleteOne-hint-unacknowledged.json 0 2 2
|
||||
deleteOne-hint.json 2 0 0
|
||||
deleteOne-let.json 0 1 1
|
||||
deleteOne-rawdata.json 1 0 1
|
||||
deleteOne.json 3 0 0
|
||||
distinct-collation.json 0 1 0
|
||||
distinct-comment.json 1 1 1
|
||||
distinct-hint.json 0 0 2
|
||||
distinct-rawdata.json 0 1 1
|
||||
distinct.json 0 2 0
|
||||
estimatedDocumentCount-comment.json 1 1 1
|
||||
estimatedDocumentCount-rawdata.json 1 0 1
|
||||
estimatedDocumentCount.json 3 1 2
|
||||
find-allowdiskuse-serverError.json 0 0 2
|
||||
find-allowdiskuse.json 3 0 0
|
||||
find-collation.json 0 1 0
|
||||
find-comment.json 1 2 2
|
||||
find-let.json 0 1 1
|
||||
find-rawdata.json 1 0 1
|
||||
find.json 2 3 0
|
||||
findOne.json 1 1 0
|
||||
findOneAndDelete-collation.json 0 1 0
|
||||
findOneAndDelete-comment.json 2 0 1
|
||||
findOneAndDelete-hint-serverError.json 0 0 2
|
||||
findOneAndDelete-hint-unacknowledged.json 0 2 2
|
||||
findOneAndDelete-hint.json 2 0 0
|
||||
findOneAndDelete-let.json 0 1 1
|
||||
findOneAndDelete-rawdata.json 1 0 1
|
||||
findOneAndDelete.json 3 0 0
|
||||
findOneAndReplace-collation.json 0 1 0
|
||||
findOneAndReplace-comment.json 0 2 1
|
||||
findOneAndReplace-dots_and_dollars.json 1 2 1
|
||||
findOneAndReplace-hint-serverError.json 0 0 2
|
||||
findOneAndReplace-hint-unacknowledged.json 0 2 2
|
||||
findOneAndReplace-hint.json 0 2 0
|
||||
findOneAndReplace-let.json 0 1 1
|
||||
findOneAndReplace-rawdata.json 0 1 1
|
||||
findOneAndReplace-upsert.json 0 4 0
|
||||
findOneAndReplace.json 2 4 0
|
||||
findOneAndUpdate-arrayFilters.json 0 3 0
|
||||
findOneAndUpdate-collation.json 0 1 0
|
||||
findOneAndUpdate-comment.json 0 2 1
|
||||
findOneAndUpdate-dots_and_dollars.json 0 0 4
|
||||
findOneAndUpdate-errorResponse.json 0 1 1
|
||||
findOneAndUpdate-hint-serverError.json 0 0 2
|
||||
findOneAndUpdate-hint-unacknowledged.json 0 2 2
|
||||
findOneAndUpdate-hint.json 2 0 0
|
||||
findOneAndUpdate-let.json 0 1 1
|
||||
findOneAndUpdate-pipeline.json 0 1 0
|
||||
findOneAndUpdate-rawdata.json 0 1 1
|
||||
findOneAndUpdate.json 5 3 0
|
||||
insertMany-comment.json 2 0 1
|
||||
insertMany-dots_and_dollars.json 0 4 1
|
||||
insertMany-rawdata.json 1 0 1
|
||||
insertMany.json 2 1 0
|
||||
insertOne-comment.json 2 0 1
|
||||
insertOne-dots_and_dollars.json 5 3 1
|
||||
insertOne-errorResponse.json 0 0 1
|
||||
insertOne-rawdata.json 1 0 1
|
||||
insertOne.json 1 0 0
|
||||
replaceOne-collation.json 0 1 0
|
||||
replaceOne-comment.json 0 2 1
|
||||
replaceOne-dots_and_dollars.json 2 2 1
|
||||
replaceOne-hint-unacknowledged.json 0 2 0
|
||||
replaceOne-hint.json 0 2 0
|
||||
replaceOne-let.json 0 1 1
|
||||
replaceOne-rawdata.json 0 1 1
|
||||
replaceOne-sort.json 1 0 1
|
||||
replaceOne-validation.json 1 0 0
|
||||
replaceOne.json 1 4 0
|
||||
updateMany-arrayFilters.json 0 3 0
|
||||
updateMany-collation.json 0 1 0
|
||||
updateMany-comment.json 2 0 1
|
||||
updateMany-dots_and_dollars.json 0 0 4
|
||||
updateMany-hint-unacknowledged.json 0 2 0
|
||||
updateMany-hint.json 2 0 0
|
||||
updateMany-let.json 0 1 1
|
||||
updateMany-pipeline.json 0 1 0
|
||||
updateMany-rawdata.json 1 0 1
|
||||
updateMany-validation.json 1 0 0
|
||||
updateMany.json 4 0 0
|
||||
updateOne-arrayFilters.json 0 5 0
|
||||
updateOne-collation.json 0 1 0
|
||||
updateOne-comment.json 2 0 1
|
||||
updateOne-dots_and_dollars.json 0 0 4
|
||||
updateOne-errorResponse.json 0 0 1
|
||||
updateOne-hint-unacknowledged.json 0 2 0
|
||||
updateOne-hint.json 2 0 0
|
||||
updateOne-let.json 0 1 1
|
||||
updateOne-pipeline.json 0 1 0
|
||||
updateOne-rawdata.json 1 0 1
|
||||
updateOne-sort.json 0 1 1
|
||||
updateOne-validation.json 1 0 0
|
||||
updateOne.json 4 0 0
|
||||
|
||||
# every non-passing case, with its reason
|
||||
aggregate-collation.json FAIL Aggregate with collation aggregate: expected 1 elements, got 0
|
||||
aggregate-let.json SKIP Aggregate with let option needs server >= 5.0
|
||||
aggregate-let.json FAIL Aggregate with let option unsupported (server-side error) aggregate: expected an error, the operation succeeded
|
||||
aggregate-let.json SKIP Aggregate to collection with let option needs server >= 5.0
|
||||
aggregate-let.json FAIL Aggregate to collection with let option unsupported (server-side error) aggregate: error message "Unrecognized pipeline stage name: '$out'" does not contain "unrecognized field 'let'"
|
||||
aggregate-merge-errorResponse.json SKIP aggregate $merge DuplicateKey error is accessible needs server >= 5.1
|
||||
aggregate-merge.json FAIL Aggregate with $merge MongoServerError: Unrecognized pipeline stage name: '$merge'
|
||||
aggregate-merge.json FAIL Aggregate with $merge and batch size of 0 MongoServerError: Unrecognized pipeline stage name: '$merge'
|
||||
aggregate-merge.json FAIL Aggregate with $merge and majority readConcern MongoServerError: Unrecognized pipeline stage name: '$merge'
|
||||
aggregate-merge.json FAIL Aggregate with $merge and local readConcern MongoServerError: Unrecognized pipeline stage name: '$merge'
|
||||
aggregate-merge.json FAIL Aggregate with $merge and available readConcern MongoServerError: Unrecognized pipeline stage name: '$merge'
|
||||
aggregate-out-readConcern.json SKIP * needs topology replicaset/sharded
|
||||
aggregate-out.json FAIL Aggregate with $out MongoServerError: Unrecognized pipeline stage name: '$out'
|
||||
aggregate-out.json FAIL Aggregate with $out and batch size of 0 MongoServerError: Unrecognized pipeline stage name: '$out'
|
||||
aggregate-rawdata.json SKIP Aggregate with rawData option needs server >= 8.2.0
|
||||
aggregate-write-readPreference.json SKIP * needs topology replicaset/sharded/load-balanced
|
||||
aggregate.json SKIP aggregate with a document comment - pre 4.4 needs server <= 4.2.99
|
||||
aggregate.json SKIP aggregate with comment does not set comment on getMore - pre 4.4 needs server <= 4.3.99
|
||||
bulkWrite-arrayFilters.json FAIL BulkWrite updateOne with arrayFilters outcome crud-tests.test[0].y: expected an array, got {"$[i]":{"b":2}}
|
||||
bulkWrite-arrayFilters.json FAIL BulkWrite updateMany with arrayFilters outcome crud-tests.test[0].y: expected an array, got {"$[i]":{"b":2}}
|
||||
bulkWrite-arrayFilters.json FAIL BulkWrite with arrayFilters outcome crud-tests.test[0].y: expected an array, got {"$[i]":{"b":2}}
|
||||
bulkWrite-collation.json FAIL BulkWrite with delete operations and collation bulkWrite.deletedCount: expected 4, got 0
|
||||
bulkWrite-collation.json FAIL BulkWrite with update operations and collation MongoBulkWriteError: internal error
|
||||
bulkWrite-comment.json FAIL BulkWrite with string comment MongoBulkWriteError: bad update
|
||||
bulkWrite-comment.json FAIL BulkWrite with document comment MongoBulkWriteError: bad update
|
||||
bulkWrite-comment.json SKIP BulkWrite with comment - pre 4.4 needs server <= 4.2.99
|
||||
bulkWrite-delete-hint-serverError.json SKIP * needs server <= 4.3.3
|
||||
bulkWrite-deleteMany-hint-unacknowledged.json SKIP Unacknowledged deleteMany with hint string fails with client-side error on pre-4.4 server needs server <= 4.2.99
|
||||
bulkWrite-deleteMany-hint-unacknowledged.json SKIP Unacknowledged deleteMany with hint document fails with client-side error on pre-4.4 server needs server <= 4.2.99
|
||||
bulkWrite-deleteMany-hint-unacknowledged.json FAIL Unacknowledged deleteMany with hint string on 4.4+ server bulkWrite: unexpected extra keys ["insertedCount","matchedCount","modifiedCount","deletedCount","upsertedCount","upsertedIds","insertedIds"]
|
||||
bulkWrite-deleteMany-hint-unacknowledged.json FAIL Unacknowledged deleteMany with hint document on 4.4+ server bulkWrite: unexpected extra keys ["insertedCount","matchedCount","modifiedCount","deletedCount","upsertedCount","upsertedIds","insertedIds"]
|
||||
bulkWrite-deleteMany-let.json SKIP BulkWrite deleteMany with let option needs server >= 5.0
|
||||
bulkWrite-deleteMany-let.json FAIL BulkWrite deleteMany with let option unsupported (server-side error) bulkWrite: expected an error, the operation succeeded
|
||||
bulkWrite-deleteMany-rawdata.json SKIP BulkWrite deleteMany with rawData option needs server >= 8.2.0
|
||||
bulkWrite-deleteOne-hint-unacknowledged.json SKIP Unacknowledged deleteOne with hint string fails with client-side error on pre-4.4 server needs server <= 4.2.99
|
||||
bulkWrite-deleteOne-hint-unacknowledged.json SKIP Unacknowledged deleteOne with hint document fails with client-side error on pre-4.4 server needs server <= 4.2.99
|
||||
bulkWrite-deleteOne-hint-unacknowledged.json FAIL Unacknowledged deleteOne with hint string on 4.4+ server bulkWrite: unexpected extra keys ["insertedCount","matchedCount","modifiedCount","deletedCount","upsertedCount","upsertedIds","insertedIds"]
|
||||
bulkWrite-deleteOne-hint-unacknowledged.json FAIL Unacknowledged deleteOne with hint document on 4.4+ server bulkWrite: unexpected extra keys ["insertedCount","matchedCount","modifiedCount","deletedCount","upsertedCount","upsertedIds","insertedIds"]
|
||||
bulkWrite-deleteOne-let.json SKIP BulkWrite deleteOne with let option needs server >= 5.0
|
||||
bulkWrite-deleteOne-let.json FAIL BulkWrite deleteOne with let option unsupported (server-side error) bulkWrite: expected an error, the operation succeeded
|
||||
bulkWrite-deleteOne-rawdata.json SKIP BulkWrite deleteOne with rawData option needs server >= 8.2.0
|
||||
bulkWrite-errorResponse.json SKIP bulkWrite operations support errorResponse assertions runner: failPoint
|
||||
bulkWrite-insertOne-dots_and_dollars.json SKIP Inserting document with top-level dollar-prefixed key on 5.0+ server needs server >= 5.0
|
||||
bulkWrite-insertOne-dots_and_dollars.json FAIL Inserting document with top-level dollar-prefixed key on pre-5.0 server yields server-side error bulkWrite: expected an error, the operation succeeded
|
||||
bulkWrite-replaceOne-dots_and_dollars.json FAIL Replacing document with top-level dotted key on 3.6+ server MongoBulkWriteError: bad update
|
||||
bulkWrite-replaceOne-dots_and_dollars.json SKIP Replacing document with dollar-prefixed key in embedded doc on 5.0+ server needs server >= 5.0
|
||||
bulkWrite-replaceOne-dots_and_dollars.json FAIL Replacing document with dotted key in embedded doc on 3.6+ server MongoBulkWriteError: bad update
|
||||
bulkWrite-replaceOne-hint-unacknowledged.json FAIL Unacknowledged replaceOne with hint string on 4.2+ server MongoBulkWriteError: bad update
|
||||
bulkWrite-replaceOne-hint-unacknowledged.json FAIL Unacknowledged replaceOne with hint document on 4.2+ server MongoBulkWriteError: bad update
|
||||
bulkWrite-replaceOne-let.json SKIP BulkWrite replaceOne with let option needs server >= 5.0
|
||||
bulkWrite-replaceOne-let.json FAIL BulkWrite replaceOne with let option unsupported (server-side error) bulkWrite: expected an error, the operation succeeded
|
||||
bulkWrite-replaceOne-rawdata.json SKIP BulkWrite replaceOne with rawData option needs server >= 8.2.0
|
||||
bulkWrite-replaceOne-sort.json SKIP BulkWrite replaceOne with sort option needs server >= 8.0
|
||||
bulkWrite-update-hint.json FAIL BulkWrite replaceOne with update hints MongoBulkWriteError: bad update
|
||||
bulkWrite-updateMany-dots_and_dollars.json SKIP Updating document to set top-level dollar-prefixed key on 5.0+ server needs server >= 5.0
|
||||
bulkWrite-updateMany-dots_and_dollars.json SKIP Updating document to set top-level dotted key on 5.0+ server needs server >= 5.0
|
||||
bulkWrite-updateMany-dots_and_dollars.json SKIP Updating document to set dollar-prefixed key in embedded doc on 5.0+ server needs server >= 5.0
|
||||
bulkWrite-updateMany-dots_and_dollars.json SKIP Updating document to set dotted key in embedded doc on 5.0+ server needs server >= 5.0
|
||||
bulkWrite-updateMany-hint-unacknowledged.json FAIL Unacknowledged updateMany with hint string on 4.2+ server bulkWrite: unexpected extra keys ["insertedCount","matchedCount","modifiedCount","deletedCount","upsertedCount","upsertedIds","insertedIds"]
|
||||
bulkWrite-updateMany-hint-unacknowledged.json FAIL Unacknowledged updateMany with hint document on 4.2+ server bulkWrite: unexpected extra keys ["insertedCount","matchedCount","modifiedCount","deletedCount","upsertedCount","upsertedIds","insertedIds"]
|
||||
bulkWrite-updateMany-let.json SKIP BulkWrite updateMany with let option needs server >= 5.0
|
||||
bulkWrite-updateMany-let.json FAIL BulkWrite updateMany with let option unsupported (server-side error) bulkWrite: error message "update spec requires u" does not contain "'update.let' is an unknown field"
|
||||
bulkWrite-updateMany-pipeline.json FAIL UpdateMany in bulk write using pipelines MongoBulkWriteError: update spec requires u
|
||||
bulkWrite-updateMany-rawdata.json SKIP BulkWrite updateMany with rawData option needs server >= 8.2.0
|
||||
bulkWrite-updateMany-rawdata.json FAIL BulkWrite updateMany with rawData option on less than 8.2.0 - ignore argument MongoBulkWriteError: update spec requires u
|
||||
bulkWrite-updateOne-dots_and_dollars.json SKIP Updating document to set top-level dollar-prefixed key on 5.0+ server needs server >= 5.0
|
||||
bulkWrite-updateOne-dots_and_dollars.json SKIP Updating document to set top-level dotted key on 5.0+ server needs server >= 5.0
|
||||
bulkWrite-updateOne-dots_and_dollars.json SKIP Updating document to set dollar-prefixed key in embedded doc on 5.0+ server needs server >= 5.0
|
||||
bulkWrite-updateOne-dots_and_dollars.json SKIP Updating document to set dotted key in embedded doc on 5.0+ server needs server >= 5.0
|
||||
bulkWrite-updateOne-hint-unacknowledged.json FAIL Unacknowledged updateOne with hint string on 4.2+ server bulkWrite: unexpected extra keys ["insertedCount","matchedCount","modifiedCount","deletedCount","upsertedCount","upsertedIds","insertedIds"]
|
||||
bulkWrite-updateOne-hint-unacknowledged.json FAIL Unacknowledged updateOne with hint document on 4.2+ server bulkWrite: unexpected extra keys ["insertedCount","matchedCount","modifiedCount","deletedCount","upsertedCount","upsertedIds","insertedIds"]
|
||||
bulkWrite-updateOne-let.json SKIP BulkWrite updateOne with let option needs server >= 5.0
|
||||
bulkWrite-updateOne-let.json FAIL BulkWrite updateOne with let option unsupported (server-side error) bulkWrite: error message "update spec requires u" does not contain "'update.let' is an unknown field"
|
||||
bulkWrite-updateOne-pipeline.json FAIL UpdateOne in bulk write using pipelines MongoBulkWriteError: update spec requires u
|
||||
bulkWrite-updateOne-rawdata.json SKIP BulkWrite updateOne with rawData option needs server >= 8.2.0
|
||||
bulkWrite-updateOne-rawdata.json FAIL BulkWrite updateOne with rawData option on less than 8.2.0 - ignore argument MongoBulkWriteError: update spec requires u
|
||||
bulkWrite-updateOne-sort.json SKIP BulkWrite updateOne with sort option needs server >= 8.0
|
||||
bulkWrite.json FAIL BulkWrite with replaceOne operations MongoBulkWriteError: bad update
|
||||
bulkWrite.json FAIL BulkWrite with updateOne operations bulkWrite.modifiedCount: expected 1, got 2
|
||||
bulkWrite.json FAIL BulkWrite with updateMany operations bulkWrite.modifiedCount: expected 2, got 4
|
||||
bulkWrite.json FAIL BulkWrite with mixed ordered operations MongoBulkWriteError: internal error
|
||||
bulkWrite.json FAIL BulkWrite with mixed unordered operations MongoBulkWriteError: internal error
|
||||
bypassDocumentValidation.json FAIL Aggregate with $out passes bypassDocumentValidation: false MongoServerError: Unrecognized pipeline stage name: '$out'
|
||||
bypassDocumentValidation.json FAIL FindOneAndReplace passes bypassDocumentValidation: false MongoServerError: bad update
|
||||
bypassDocumentValidation.json FAIL ReplaceOne passes bypassDocumentValidation: false MongoServerError: bad update
|
||||
client-bulkWrite-delete-options.json SKIP * needs server >= 8.0
|
||||
client-bulkWrite-delete-rawdata.json SKIP client bulk write delete with rawData option needs server >= 8.2.0
|
||||
client-bulkWrite-delete-rawdata.json SKIP client bulk write delete with rawData option on less than 8.2.0 - ignore argument needs server >= 8.0
|
||||
client-bulkWrite-errorResponse.json SKIP * needs server >= 8.0
|
||||
client-bulkWrite-errors.json SKIP * needs server >= 8.0
|
||||
client-bulkWrite-mixed-namespaces.json SKIP * needs server >= 8.0
|
||||
client-bulkWrite-options.json SKIP * needs server >= 8.0
|
||||
client-bulkWrite-ordered.json SKIP * needs server >= 8.0
|
||||
client-bulkWrite-partialResults.json SKIP * needs server >= 8.0
|
||||
client-bulkWrite-replaceOne-rawdata.json SKIP client bulkWrite replaceOne with rawData option needs server >= 8.2.0
|
||||
client-bulkWrite-replaceOne-rawdata.json SKIP client bulk write replaceOne with rawData option on less than 8.2.0 - ignore argument needs server >= 8.0
|
||||
client-bulkWrite-replaceOne-sort.json SKIP * needs server >= 8.0
|
||||
client-bulkWrite-results.json SKIP * needs server >= 8.0
|
||||
client-bulkWrite-update-options.json SKIP * needs server >= 8.0
|
||||
client-bulkWrite-update-pipeline.json SKIP * needs server >= 8.0
|
||||
client-bulkWrite-update-rawdata.json SKIP client bulk write update with rawData option needs server >= 8.2.0
|
||||
client-bulkWrite-update-rawdata.json SKIP client bulk write update with rawData option on less than 8.2.0 - ignore argument needs server >= 8.0
|
||||
client-bulkWrite-update-validation.json SKIP client bulkWrite replaceOne prohibits atomic modifiers runner: operation clientBulkWrite
|
||||
client-bulkWrite-update-validation.json SKIP client bulkWrite updateOne requires atomic modifiers runner: operation clientBulkWrite
|
||||
client-bulkWrite-update-validation.json SKIP client bulkWrite updateMany requires atomic modifiers runner: operation clientBulkWrite
|
||||
client-bulkWrite-updateOne-sort.json SKIP * needs server >= 8.0
|
||||
count-collation.json SKIP Deprecated count with collation runner: operation count
|
||||
count-empty.json SKIP Deprecated count with empty collection runner: operation count
|
||||
count-rawdata.json SKIP Deprecated count with rawData option needs server >= 8.2.0
|
||||
count-rawdata.json SKIP Deprecated count with rawData option on less than 8.2.0 - ignore argument runner: operation count
|
||||
count.json FAIL Count documents with skip and limit countDocuments: expected 2, got 3
|
||||
count.json SKIP Deprecated count without a filter runner: operation count
|
||||
count.json SKIP Deprecated count with a filter runner: operation count
|
||||
count.json SKIP Deprecated count with skip and limit runner: operation count
|
||||
countDocuments-comment.json SKIP countDocuments with document comment on less than 4.4.0 - server error needs server <= 4.3.99
|
||||
countDocuments-rawdata.json SKIP Count documents with rawData option needs server >= 8.2.0
|
||||
create-null-ids.json FAIL inserting _id with type null via insertOne countDocuments: expected 1, got 0
|
||||
create-null-ids.json FAIL inserting _id with type null via insertMany countDocuments: expected 1, got 0
|
||||
create-null-ids.json FAIL inserting _id with type null via updateOne countDocuments: expected 1, got 0
|
||||
create-null-ids.json FAIL inserting _id with type null via updateMany countDocuments: expected 1, got 0
|
||||
create-null-ids.json FAIL inserting _id with type null via replaceOne MongoServerError: internal error
|
||||
create-null-ids.json FAIL inserting _id with type null via bulkWrite countDocuments: expected 1, got 0
|
||||
create-null-ids.json SKIP inserting _id with type null via clientBulkWrite needs server >= 8.0
|
||||
db-aggregate-rawdata.json SKIP Aggregate with rawData option needs server >= 8.2.0, needs topology replicaset
|
||||
db-aggregate-rawdata.json FAIL Aggregate with rawData option on less than 8.2.0 - ignore argument MongoServerError: command requires a collection name
|
||||
db-aggregate-write-readPreference.json SKIP * needs topology replicaset
|
||||
db-aggregate.json FAIL Aggregate with $listLocalSessions MongoServerError: command requires a collection name
|
||||
db-aggregate.json FAIL Aggregate with $listLocalSessions and allowDiskUse MongoServerError: command requires a collection name
|
||||
deleteMany-collation.json FAIL DeleteMany when many documents match with collation deleteMany.deletedCount: expected 2, got 0
|
||||
deleteMany-comment.json SKIP deleteMany with comment - pre 4.4 needs server <= 4.2.99
|
||||
deleteMany-hint-serverError.json SKIP * needs server <= 4.3.3
|
||||
deleteMany-hint-unacknowledged.json SKIP Unacknowledged deleteMany with hint string fails with client-side error on pre-4.4 server needs server <= 4.2.99
|
||||
deleteMany-hint-unacknowledged.json SKIP Unacknowledged deleteMany with hint document fails with client-side error on pre-4.4 server needs server <= 4.2.99
|
||||
deleteMany-hint-unacknowledged.json FAIL Unacknowledged deleteMany with hint string on 4.4+ server deleteMany: unexpected extra keys ["deletedCount"]
|
||||
deleteMany-hint-unacknowledged.json FAIL Unacknowledged deleteMany with hint document on 4.4+ server deleteMany: unexpected extra keys ["deletedCount"]
|
||||
deleteMany-let.json SKIP deleteMany with let option needs server >= 5.0
|
||||
deleteMany-let.json FAIL deleteMany with let option unsupported (server-side error) deleteMany: expected an error, the operation succeeded
|
||||
deleteMany-rawdata.json SKIP deleteMany with rawData option needs server >= 8.2.0
|
||||
deleteOne-collation.json FAIL DeleteOne when many documents matches with collation deleteOne.deletedCount: expected 1, got 0
|
||||
deleteOne-comment.json SKIP deleteOne with comment - pre 4.4 needs server <= 4.2.99
|
||||
deleteOne-errorResponse.json SKIP delete operations support errorResponse assertions runner: failPoint
|
||||
deleteOne-hint-serverError.json SKIP * needs server <= 4.3.3
|
||||
deleteOne-hint-unacknowledged.json SKIP Unacknowledged deleteOne with hint string fails with client-side error on pre-4.4 server needs server <= 4.2.99
|
||||
deleteOne-hint-unacknowledged.json SKIP Unacknowledged deleteOne with hint document fails with client-side error on pre-4.4 server needs server <= 4.2.99
|
||||
deleteOne-hint-unacknowledged.json FAIL Unacknowledged deleteOne with hint string on 4.4+ server deleteOne: unexpected extra keys ["deletedCount"]
|
||||
deleteOne-hint-unacknowledged.json FAIL Unacknowledged deleteOne with hint document on 4.4+ server deleteOne: unexpected extra keys ["deletedCount"]
|
||||
deleteOne-let.json SKIP deleteOne with let option needs server >= 5.0
|
||||
deleteOne-let.json FAIL deleteOne with let option unsupported (server-side error) deleteOne: expected an error, the operation succeeded
|
||||
deleteOne-rawdata.json SKIP deleteOne with rawData option needs server >= 8.2.0
|
||||
distinct-collation.json FAIL Distinct with a collation MongoServerError: no such command: 'distinct'
|
||||
distinct-comment.json SKIP distinct with document comment needs server >= 4.4.14
|
||||
distinct-comment.json FAIL distinct with string comment MongoServerError: no such command: 'distinct'
|
||||
distinct-hint.json SKIP * needs server >= 7.1.0
|
||||
distinct-rawdata.json SKIP distinct with rawData option needs server >= 8.2.0
|
||||
distinct-rawdata.json FAIL distinct with rawData option on less than 8.2.0 - ignore argument MongoServerError: no such command: 'distinct'
|
||||
distinct.json FAIL Distinct without a filter MongoServerError: no such command: 'distinct'
|
||||
distinct.json FAIL Distinct with a filter MongoServerError: no such command: 'distinct'
|
||||
estimatedDocumentCount-comment.json SKIP estimatedDocumentCount with document comment needs server >= 4.4.14
|
||||
estimatedDocumentCount-comment.json FAIL estimatedDocumentCount with document comment - pre 4.4.14, server error estimatedDocumentCount: expected an error, the operation succeeded
|
||||
estimatedDocumentCount-rawdata.json SKIP Estimated document count with rawData option needs server >= 8.2.0
|
||||
estimatedDocumentCount.json SKIP estimatedDocumentCount errors correctly--command error runner: failPoint
|
||||
estimatedDocumentCount.json SKIP estimatedDocumentCount errors correctly--socket error runner: failPoint
|
||||
estimatedDocumentCount.json FAIL estimatedDocumentCount works correctly on views estimatedDocumentCount: expected 2, got 0
|
||||
find-allowdiskuse-serverError.json SKIP * needs server <= 4.3.0
|
||||
find-collation.json FAIL Find with a collation find: expected 1 elements, got 0
|
||||
find-comment.json FAIL find with string comment find[0]: unexpected extra keys ["x"]
|
||||
find-comment.json FAIL find with document comment find[0]: unexpected extra keys ["x"]
|
||||
find-comment.json SKIP find with document comment - pre 4.4 needs server <= 4.2.99
|
||||
find-comment.json SKIP find with comment does not set comment on getMore - pre 4.4 needs server <= 4.3.99
|
||||
find-let.json SKIP Find with let option needs server >= 5.0
|
||||
find-let.json FAIL Find with let option unsupported (server-side error) find: expected an error, the operation succeeded
|
||||
find-rawdata.json SKIP Find with rawData option needs server >= 8.2.0
|
||||
find.json FAIL Find with filter, sort, skip, and limit find: expected 2 elements, got 4
|
||||
find.json FAIL Find with limit, sort, and batchsize find: expected 4 elements, got 6
|
||||
find.json FAIL Find with batchSize equal to limit find: expected 4 elements, got 5
|
||||
findOne.json FAIL FindOne with filter, sort, and skip findOne._id: expected 5, got 3
|
||||
findOneAndDelete-collation.json FAIL FindOneAndDelete when one document matches with collation findOneAndDelete: expected a document, got null
|
||||
findOneAndDelete-comment.json SKIP findOneAndDelete with comment - pre 4.4 needs server <= 4.2.99
|
||||
findOneAndDelete-hint-serverError.json SKIP * needs server <= 4.3.3
|
||||
findOneAndDelete-hint-unacknowledged.json SKIP Unacknowledged findOneAndDelete with hint string fails with client-side error on pre-4.4 server needs server <= 4.2.99
|
||||
findOneAndDelete-hint-unacknowledged.json SKIP Unacknowledged findOneAndDelete with hint document fails with client-side error on pre-4.4 server needs server <= 4.2.99
|
||||
findOneAndDelete-hint-unacknowledged.json FAIL Unacknowledged findOneAndDelete with hint string on 4.4+ server findOneAndDelete: expected null, got {"_id":2,"x":22}
|
||||
findOneAndDelete-hint-unacknowledged.json FAIL Unacknowledged findOneAndDelete with hint document on 4.4+ server findOneAndDelete: expected null, got {"_id":2,"x":22}
|
||||
findOneAndDelete-let.json SKIP findOneAndDelete with let option needs server >= 5.0
|
||||
findOneAndDelete-let.json FAIL findOneAndDelete with let option unsupported (server-side error) findOneAndDelete: expected an error, the operation succeeded
|
||||
findOneAndDelete-rawdata.json SKIP findOneAndDelete with rawData option needs server >= 8.2.0
|
||||
findOneAndReplace-collation.json FAIL FindOneAndReplace when one document matches with collation returning the document after modification findOneAndReplace: expected a document, got null
|
||||
findOneAndReplace-comment.json FAIL findOneAndReplace with string comment MongoServerError: bad update
|
||||
findOneAndReplace-comment.json FAIL findOneAndReplace with document comment MongoServerError: bad update
|
||||
findOneAndReplace-comment.json SKIP findOneAndReplace with comment - pre 4.4 needs server <= 4.2.99
|
||||
findOneAndReplace-dots_and_dollars.json FAIL Replacing document with top-level dotted key on 3.6+ server MongoServerError: bad update
|
||||
findOneAndReplace-dots_and_dollars.json SKIP Replacing document with dollar-prefixed key in embedded doc on 5.0+ server needs server >= 5.0
|
||||
findOneAndReplace-dots_and_dollars.json FAIL Replacing document with dotted key in embedded doc on 3.6+ server MongoServerError: bad update
|
||||
findOneAndReplace-hint-serverError.json SKIP * needs server <= 4.3.0
|
||||
findOneAndReplace-hint-unacknowledged.json SKIP Unacknowledged findOneAndReplace with hint string fails with client-side error on pre-4.4 server needs server <= 4.2.99
|
||||
findOneAndReplace-hint-unacknowledged.json SKIP Unacknowledged findOneAndReplace with hint document fails with client-side error on pre-4.4 server needs server <= 4.2.99
|
||||
findOneAndReplace-hint-unacknowledged.json FAIL Unacknowledged findOneAndReplace with hint string on 4.4+ server MongoServerError: bad update
|
||||
findOneAndReplace-hint-unacknowledged.json FAIL Unacknowledged findOneAndReplace with hint document on 4.4+ server MongoServerError: bad update
|
||||
findOneAndReplace-hint.json FAIL FindOneAndReplace with hint string MongoServerError: bad update
|
||||
findOneAndReplace-hint.json FAIL FindOneAndReplace with hint document MongoServerError: bad update
|
||||
findOneAndReplace-let.json SKIP findOneAndReplace with let option needs server >= 5.0
|
||||
findOneAndReplace-let.json FAIL findOneAndReplace with let option unsupported (server-side error) findOneAndReplace: expected an error, the operation succeeded
|
||||
findOneAndReplace-rawdata.json SKIP findOneAndReplace with rawData option needs server >= 8.2.0
|
||||
findOneAndReplace-rawdata.json FAIL findOneAndReplace with rawData option on less than 8.2.0 - ignore argument MongoServerError: bad update
|
||||
findOneAndReplace-upsert.json FAIL FindOneAndReplace when no documents match without id specified with upsert returning the document before modification MongoServerError: internal error
|
||||
findOneAndReplace-upsert.json FAIL FindOneAndReplace when no documents match without id specified with upsert returning the document after modification MongoServerError: internal error
|
||||
findOneAndReplace-upsert.json FAIL FindOneAndReplace when no documents match with id specified with upsert returning the document before modification MongoServerError: internal error
|
||||
findOneAndReplace-upsert.json FAIL FindOneAndReplace when no documents match with id specified with upsert returning the document after modification MongoServerError: internal error
|
||||
findOneAndReplace.json FAIL FindOneAndReplace when many documents match returning the document before modification MongoServerError: bad update
|
||||
findOneAndReplace.json FAIL FindOneAndReplace when many documents match returning the document after modification MongoServerError: bad update
|
||||
findOneAndReplace.json FAIL FindOneAndReplace when one document matches returning the document before modification MongoServerError: bad update
|
||||
findOneAndReplace.json FAIL FindOneAndReplace when one document matches returning the document after modification MongoServerError: bad update
|
||||
findOneAndUpdate-arrayFilters.json FAIL FindOneAndUpdate when no document matches arrayFilters outcome crud-v1.coll[0].y: expected an array, got {"$[i]":{"b":2}}
|
||||
findOneAndUpdate-arrayFilters.json FAIL FindOneAndUpdate when one document matches arrayFilters outcome crud-v1.coll[0].y: expected an array, got {"$[i]":{"b":2}}
|
||||
findOneAndUpdate-arrayFilters.json FAIL FindOneAndUpdate when multiple documents match arrayFilters outcome crud-v1.coll[0].y: expected an array, got {"$[i]":{"b":2}}
|
||||
findOneAndUpdate-collation.json FAIL FindOneAndUpdate when many documents match with collation returning the document before modification findOneAndUpdate: expected a document, got null
|
||||
findOneAndUpdate-comment.json FAIL findOneAndUpdate with string comment MongoServerError: update must be a document
|
||||
findOneAndUpdate-comment.json FAIL findOneAndUpdate with document comment MongoServerError: update must be a document
|
||||
findOneAndUpdate-comment.json SKIP findOneAndUpdate with comment - pre 4.4 needs server <= 4.2.99
|
||||
findOneAndUpdate-dots_and_dollars.json SKIP Updating document to set top-level dollar-prefixed key on 5.0+ server needs server >= 5.0
|
||||
findOneAndUpdate-dots_and_dollars.json SKIP Updating document to set top-level dotted key on 5.0+ server needs server >= 5.0
|
||||
findOneAndUpdate-dots_and_dollars.json SKIP Updating document to set dollar-prefixed key in embedded doc on 5.0+ server needs server >= 5.0
|
||||
findOneAndUpdate-dots_and_dollars.json SKIP Updating document to set dotted key in embedded doc on 5.0+ server needs server >= 5.0
|
||||
findOneAndUpdate-errorResponse.json FAIL findOneAndUpdate DuplicateKey error is accessible findOneAndUpdate.errorResponse.keyPattern: missing from actual
|
||||
findOneAndUpdate-errorResponse.json SKIP findOneAndUpdate document validation errInfo is accessible needs server >= 5.0
|
||||
findOneAndUpdate-hint-serverError.json SKIP * needs server <= 4.3.0
|
||||
findOneAndUpdate-hint-unacknowledged.json SKIP Unacknowledged findOneAndUpdate with hint string fails with client-side error on pre-4.4 server needs server <= 4.2.99
|
||||
findOneAndUpdate-hint-unacknowledged.json SKIP Unacknowledged findOneAndUpdate with hint document fails with client-side error on pre-4.4 server needs server <= 4.2.99
|
||||
findOneAndUpdate-hint-unacknowledged.json FAIL Unacknowledged findOneAndUpdate with hint string on 4.4+ server findOneAndUpdate: expected null, got {"_id":2,"x":22}
|
||||
findOneAndUpdate-hint-unacknowledged.json FAIL Unacknowledged findOneAndUpdate with hint document on 4.4+ server findOneAndUpdate: expected null, got {"_id":2,"x":22}
|
||||
findOneAndUpdate-let.json SKIP findOneAndUpdate with let option needs server >= 5.0
|
||||
findOneAndUpdate-let.json FAIL findOneAndUpdate with let option unsupported (server-side error) findOneAndUpdate: expected an error, the operation succeeded
|
||||
findOneAndUpdate-pipeline.json FAIL FindOneAndUpdate using pipelines MongoServerError: update must be a document
|
||||
findOneAndUpdate-rawdata.json SKIP findOneAndUpdate with rawData option needs server >= 8.2.0
|
||||
findOneAndUpdate-rawdata.json FAIL findOneAndUpdate with rawData option on less than 8.2.0 - ignore argument MongoServerError: update must be a document
|
||||
findOneAndUpdate.json FAIL FindOneAndUpdate when many documents match returning the document after modification findOneAndUpdate.x: expected 23, got 22
|
||||
findOneAndUpdate.json FAIL FindOneAndUpdate when one document matches returning the document after modification findOneAndUpdate.x: expected 23, got 22
|
||||
findOneAndUpdate.json FAIL FindOneAndUpdate when no documents match with upsert returning the document after modification findOneAndUpdate: expected a document, got null
|
||||
insertMany-comment.json SKIP insertMany with comment - pre 4.4 needs server <= 4.2.99
|
||||
insertMany-dots_and_dollars.json SKIP Inserting document with top-level dollar-prefixed key on 5.0+ server needs server >= 5.0
|
||||
insertMany-dots_and_dollars.json FAIL Inserting document with top-level dollar-prefixed key on pre-5.0 server yields server-side error insertMany: expected an error, the operation succeeded
|
||||
insertMany-dots_and_dollars.json FAIL Inserting document with top-level dotted key insertMany: unexpected extra keys ["insertedCount"]
|
||||
insertMany-dots_and_dollars.json FAIL Inserting document with dollar-prefixed key in embedded doc insertMany: unexpected extra keys ["insertedCount"]
|
||||
insertMany-dots_and_dollars.json FAIL Inserting document with dotted key in embedded doc insertMany: unexpected extra keys ["insertedCount"]
|
||||
insertMany-rawdata.json SKIP insertMany with rawData option needs server >= 8.2.0
|
||||
insertMany.json FAIL InsertMany with non-existing documents insertMany: unexpected extra keys ["insertedCount"]
|
||||
insertOne-comment.json SKIP insertOne with comment - pre 4.4 needs server <= 4.2.99
|
||||
insertOne-dots_and_dollars.json SKIP Inserting document with top-level dollar-prefixed key on 5.0+ server needs server >= 5.0
|
||||
insertOne-dots_and_dollars.json FAIL Inserting document with top-level dollar-prefixed key on pre-5.0 server yields server-side error insertOne: expected an error, the operation succeeded
|
||||
insertOne-dots_and_dollars.json FAIL Inserting document with dollar-prefixed key in _id yields server-side error insertOne: expected an error, the operation succeeded
|
||||
insertOne-dots_and_dollars.json FAIL Unacknowledged write using dollar-prefixed or dotted keys may be silently rejected on pre-5.0 server insertOne: unexpected extra keys ["insertedId"]
|
||||
insertOne-errorResponse.json SKIP insert operations support errorResponse assertions runner: failPoint
|
||||
insertOne-rawdata.json SKIP insertOne with rawData option needs server >= 8.2.0
|
||||
replaceOne-collation.json FAIL ReplaceOne when one document matches with collation replaceOne.matchedCount: expected 1, got 0
|
||||
replaceOne-comment.json FAIL ReplaceOne with string comment MongoServerError: bad update
|
||||
replaceOne-comment.json FAIL ReplaceOne with document comment MongoServerError: bad update
|
||||
replaceOne-comment.json SKIP ReplaceOne with comment - pre 4.4 needs server <= 4.2.99
|
||||
replaceOne-dots_and_dollars.json FAIL Replacing document with top-level dotted key on 3.6+ server MongoServerError: bad update
|
||||
replaceOne-dots_and_dollars.json SKIP Replacing document with dollar-prefixed key in embedded doc on 5.0+ server needs server >= 5.0
|
||||
replaceOne-dots_and_dollars.json FAIL Replacing document with dotted key in embedded doc on 3.6+ server MongoServerError: bad update
|
||||
replaceOne-hint-unacknowledged.json FAIL Unacknowledged replaceOne with hint string on 4.2+ server MongoServerError: bad update
|
||||
replaceOne-hint-unacknowledged.json FAIL Unacknowledged replaceOne with hint document on 4.2+ server MongoServerError: bad update
|
||||
replaceOne-hint.json FAIL ReplaceOne with hint string MongoServerError: bad update
|
||||
replaceOne-hint.json FAIL ReplaceOne with hint document MongoServerError: bad update
|
||||
replaceOne-let.json SKIP ReplaceOne with let option needs server >= 5.0
|
||||
replaceOne-let.json FAIL ReplaceOne with let option unsupported (server-side error) replaceOne: expected an error, the operation succeeded
|
||||
replaceOne-rawdata.json SKIP ReplaceOne with rawData option needs server >= 8.2.0
|
||||
replaceOne-rawdata.json FAIL ReplaceOne with rawData option on less than 8.2.0 - ignore argument MongoServerError: bad update
|
||||
replaceOne-sort.json SKIP ReplaceOne with sort option needs server >= 8.0
|
||||
replaceOne.json FAIL ReplaceOne when many documents match MongoServerError: bad update
|
||||
replaceOne.json FAIL ReplaceOne when one document matches MongoServerError: bad update
|
||||
replaceOne.json FAIL ReplaceOne with upsert when no documents match without an id specified MongoServerError: internal error
|
||||
replaceOne.json FAIL ReplaceOne with upsert when no documents match with an id specified MongoServerError: internal error
|
||||
updateMany-arrayFilters.json FAIL UpdateMany when no documents match arrayFilters updateMany.modifiedCount: expected 0, got 2
|
||||
updateMany-arrayFilters.json FAIL UpdateMany when one document matches arrayFilters updateMany.modifiedCount: expected 1, got 2
|
||||
updateMany-arrayFilters.json FAIL UpdateMany when multiple documents match arrayFilters outcome crud-v1.coll[0].y: expected an array, got {"$[i]":{"b":2}}
|
||||
updateMany-collation.json FAIL UpdateMany when many documents match with collation updateMany.matchedCount: expected 2, got 1
|
||||
updateMany-comment.json SKIP UpdateMany with comment - pre 4.4 needs server <= 4.2.99
|
||||
updateMany-dots_and_dollars.json SKIP Updating document to set top-level dollar-prefixed key on 5.0+ server needs server >= 5.0
|
||||
updateMany-dots_and_dollars.json SKIP Updating document to set top-level dotted key on 5.0+ server needs server >= 5.0
|
||||
updateMany-dots_and_dollars.json SKIP Updating document to set dollar-prefixed key in embedded doc on 5.0+ server needs server >= 5.0
|
||||
updateMany-dots_and_dollars.json SKIP Updating document to set dotted key in embedded doc on 5.0+ server needs server >= 5.0
|
||||
updateMany-hint-unacknowledged.json FAIL Unacknowledged updateMany with hint string on 4.2+ server updateMany: unexpected extra keys ["modifiedCount","upsertedId","upsertedCount","matchedCount"]
|
||||
updateMany-hint-unacknowledged.json FAIL Unacknowledged updateMany with hint document on 4.2+ server updateMany: unexpected extra keys ["modifiedCount","upsertedId","upsertedCount","matchedCount"]
|
||||
updateMany-let.json SKIP updateMany with let option needs server >= 5.0
|
||||
updateMany-let.json FAIL updateMany with let option unsupported (server-side error) updateMany: error message "update spec requires u" does not contain "'update.let' is an unknown field"
|
||||
updateMany-pipeline.json FAIL UpdateMany using pipelines MongoServerError: update spec requires u
|
||||
updateMany-rawdata.json SKIP updateMany with rawData option needs server >= 8.2.0
|
||||
updateOne-arrayFilters.json FAIL UpdateOne when no document matches arrayFilters updateOne.modifiedCount: expected 0, got 1
|
||||
updateOne-arrayFilters.json FAIL UpdateOne when one document matches arrayFilters outcome crud-v1.coll[0].y: expected an array, got {"$[i]":{"b":2}}
|
||||
updateOne-arrayFilters.json FAIL UpdateOne when multiple documents match arrayFilters outcome crud-v1.coll[0].y: expected an array, got {"$[i]":{"b":2}}
|
||||
updateOne-arrayFilters.json FAIL UpdateOne when no documents match multiple arrayFilters updateOne.modifiedCount: expected 0, got 1
|
||||
updateOne-arrayFilters.json FAIL UpdateOne when one document matches multiple arrayFilters outcome crud-v1.coll[2].y: expected an array, got {"$[i]":{"c":{"$[j]":{"d":0}}}}
|
||||
updateOne-collation.json FAIL UpdateOne when one document matches with collation updateOne.matchedCount: expected 1, got 0
|
||||
updateOne-comment.json SKIP UpdateOne with comment - pre 4.4 needs server <= 4.2.99
|
||||
updateOne-dots_and_dollars.json SKIP Updating document to set top-level dollar-prefixed key on 5.0+ server needs server >= 5.0
|
||||
updateOne-dots_and_dollars.json SKIP Updating document to set top-level dotted key on 5.0+ server needs server >= 5.0
|
||||
updateOne-dots_and_dollars.json SKIP Updating document to set dollar-prefixed key in embedded doc on 5.0+ server needs server >= 5.0
|
||||
updateOne-dots_and_dollars.json SKIP Updating document to set dotted key in embedded doc on 5.0+ server needs server >= 5.0
|
||||
updateOne-errorResponse.json SKIP update operations support errorResponse assertions runner: failPoint
|
||||
updateOne-hint-unacknowledged.json FAIL Unacknowledged updateOne with hint string on 4.2+ server updateOne: unexpected extra keys ["modifiedCount","upsertedId","upsertedCount","matchedCount"]
|
||||
updateOne-hint-unacknowledged.json FAIL Unacknowledged updateOne with hint document on 4.2+ server updateOne: unexpected extra keys ["modifiedCount","upsertedId","upsertedCount","matchedCount"]
|
||||
updateOne-let.json SKIP UpdateOne with let option needs server >= 5.0
|
||||
updateOne-let.json FAIL UpdateOne with let option unsupported (server-side error) updateOne: error message "update spec requires u" does not contain "'update.let' is an unknown field"
|
||||
updateOne-pipeline.json FAIL UpdateOne using pipelines MongoServerError: update spec requires u
|
||||
updateOne-rawdata.json SKIP UpdateOne with rawData option needs server >= 8.2.0
|
||||
updateOne-sort.json SKIP UpdateOne with sort option needs server >= 8.0
|
||||
updateOne-sort.json FAIL updateOne with sort option unsupported (server-side error) updateOne: expected an error, the operation succeeded
|
||||
Reference in New Issue
Block a user