tests/spec: MongoDB spec-test runner and the M0 scorecard
PLAN D2 makes the official specification suites the gate for command semantics; D7.6 asks for the harness to exist at M0 with a recorded baseline. This is that harness, pinned on both sides -- mongodb/specifications @ 615e0f9 and mongodb@7.5.0 -- because a scorecard is only comparable across milestones if a delta cannot be an upstream test change. It implements the unified format's Evaluating Matches algorithm as written, including the two rules that decide whether a pass is earned: extra keys are tolerated only in a root document, and numeric types compare flexibly. Anything unimplemented is a SKIP with a reason, never a pass, and the one assertion class not yet checked -- expectEvents, i.e. command monitoring -- is disclosed at the top of the scorecard so `pass` reads as an upper bound. First honest run: 131 pass, 161 fail, 195 skip over 175 files, zero timeouts. Getting there took four attempts, and the failures are documented in the README because each would have shipped a scorecard claiming a compatibility gap that did not exist. Two were genuine leaks in this runner (clients left open when a case timed out; clients registered for cleanup only after `await connect()`, plus abandoned cases still creating more). The third I misdiagnosed as machine load. The fourth attempt found the real cause: a leaked catalog lock in the engine, fixed separately, which alone accounts for the jump from 45 passes to 131. So the runner carries its own guards: per-operation CSOT timeouts so work is never abandoned, an active-handle census per file, an end-of-run tripwire for stray timers, a hard stop if the server dies rather than emitting hundreds of misleading ECONNREFUSED failures, and --skip/--limit for bisecting a run whose failures depend on position. The README states the rule plainly -- a long unbroken tail of timeouts is a harness bug until proven otherwise -- and the two commands that settle it. Also fixes bench-run.sh, which copied its report over bench-latest.txt unconditionally, including after a run that only warned -- so a degraded run could silently replace the baseline that PLAN D7.5 makes a milestone gate.
This commit is contained in:
44
tests/spec/fetch.sh
Executable file
44
tests/spec/fetch.sh
Executable file
@@ -0,0 +1,44 @@
|
||||
#!/usr/bin/env bash
|
||||
# Fetch the pinned mongodb/specifications test suites.
|
||||
#
|
||||
# Pinned, not vendored: the suites are ~175 JSON files that upstream rewrites
|
||||
# wholesale, and a pinned commit gives the same reproducibility as vendoring
|
||||
# without putting someone else's test corpus in this repo's history. PLAN D2
|
||||
# requires the pin; `specifications/` is gitignored.
|
||||
#
|
||||
# Bump SPEC_COMMIT deliberately, in its own commit, and re-record the
|
||||
# scorecard in the same commit -- otherwise a scorecard delta cannot be told
|
||||
# apart from an upstream change.
|
||||
set -euo pipefail
|
||||
|
||||
SPEC_COMMIT="615e0f9ca5f554614636c098225fbbf1be55565d"
|
||||
SPEC_REPO="https://github.com/mongodb/specifications.git"
|
||||
|
||||
cd "$(dirname "$0")"
|
||||
DEST="specifications"
|
||||
|
||||
if [ -d "$DEST/.git" ] && [ "$(git -C "$DEST" rev-parse HEAD 2>/dev/null || true)" = "$SPEC_COMMIT" ]; then
|
||||
echo "specs already at $SPEC_COMMIT"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
rm -rf "$DEST"
|
||||
mkdir -p "$DEST"
|
||||
cd "$DEST"
|
||||
|
||||
# Sparse + depth 1: the full history is ~100 MB and we need two directories.
|
||||
git init -q .
|
||||
git remote add origin "$SPEC_REPO"
|
||||
git config core.sparseCheckout true
|
||||
mkdir -p .git/info
|
||||
cat > .git/info/sparse-checkout <<'EOF'
|
||||
source/crud/tests/
|
||||
source/unified-test-format/
|
||||
EOF
|
||||
|
||||
echo "fetching $SPEC_COMMIT ..."
|
||||
git fetch -q --depth 1 origin "$SPEC_COMMIT"
|
||||
git checkout -q FETCH_HEAD
|
||||
|
||||
n=$(find source/crud/tests/unified -name '*.json' | wc -l | tr -d ' ')
|
||||
echo "specs at $SPEC_COMMIT ($n crud/unified json files)"
|
||||
Reference in New Issue
Block a user