Aleksey Shakhmatov
90de7820da
PLAN D2 makes the official specification suites the gate for command semantics; D7.6 asks for the harness to exist at M0 with a recorded baseline. This is that harness, pinned on both sides -- mongodb/specifications @ 615e0f9 and mongodb@7.5.0 -- because a scorecard is only comparable across milestones if a delta cannot be an upstream test change. It implements the unified format's Evaluating Matches algorithm as written, including the two rules that decide whether a pass is earned: extra keys are tolerated only in a root document, and numeric types compare flexibly. Anything unimplemented is a SKIP with a reason, never a pass, and the one assertion class not yet checked -- expectEvents, i.e. command monitoring -- is disclosed at the top of the scorecard so `pass` reads as an upper bound. First honest run: 131 pass, 161 fail, 195 skip over 175 files, zero timeouts. Getting there took four attempts, and the failures are documented in the README because each would have shipped a scorecard claiming a compatibility gap that did not exist. Two were genuine leaks in this runner (clients left open when a case timed out; clients registered for cleanup only after `await connect()`, plus abandoned cases still creating more). The third I misdiagnosed as machine load. The fourth attempt found the real cause: a leaked catalog lock in the engine, fixed separately, which alone accounts for the jump from 45 passes to 131. So the runner carries its own guards: per-operation CSOT timeouts so work is never abandoned, an active-handle census per file, an end-of-run tripwire for stray timers, a hard stop if the server dies rather than emitting hundreds of misleading ECONNREFUSED failures, and --skip/--limit for bisecting a run whose failures depend on position. The README states the rule plainly -- a long unbroken tail of timeouts is a harness bug until proven otherwise -- and the two commands that settle it. Also fixes bench-run.sh, which copied its report over bench-latest.txt unconditionally, including after a run that only warned -- so a degraded run could silently replace the baseline that PLAN D7.5 makes a milestone gate.
45 lines
1.4 KiB
Bash
Executable File
45 lines
1.4 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Fetch the pinned mongodb/specifications test suites.
|
|
#
|
|
# Pinned, not vendored: the suites are ~175 JSON files that upstream rewrites
|
|
# wholesale, and a pinned commit gives the same reproducibility as vendoring
|
|
# without putting someone else's test corpus in this repo's history. PLAN D2
|
|
# requires the pin; `specifications/` is gitignored.
|
|
#
|
|
# Bump SPEC_COMMIT deliberately, in its own commit, and re-record the
|
|
# scorecard in the same commit -- otherwise a scorecard delta cannot be told
|
|
# apart from an upstream change.
|
|
set -euo pipefail
|
|
|
|
SPEC_COMMIT="615e0f9ca5f554614636c098225fbbf1be55565d"
|
|
SPEC_REPO="https://github.com/mongodb/specifications.git"
|
|
|
|
cd "$(dirname "$0")"
|
|
DEST="specifications"
|
|
|
|
if [ -d "$DEST/.git" ] && [ "$(git -C "$DEST" rev-parse HEAD 2>/dev/null || true)" = "$SPEC_COMMIT" ]; then
|
|
echo "specs already at $SPEC_COMMIT"
|
|
exit 0
|
|
fi
|
|
|
|
rm -rf "$DEST"
|
|
mkdir -p "$DEST"
|
|
cd "$DEST"
|
|
|
|
# Sparse + depth 1: the full history is ~100 MB and we need two directories.
|
|
git init -q .
|
|
git remote add origin "$SPEC_REPO"
|
|
git config core.sparseCheckout true
|
|
mkdir -p .git/info
|
|
cat > .git/info/sparse-checkout <<'EOF'
|
|
source/crud/tests/
|
|
source/unified-test-format/
|
|
EOF
|
|
|
|
echo "fetching $SPEC_COMMIT ..."
|
|
git fetch -q --depth 1 origin "$SPEC_COMMIT"
|
|
git checkout -q FETCH_HEAD
|
|
|
|
n=$(find source/crud/tests/unified -name '*.json' | wc -l | tr -d ' ')
|
|
echo "specs at $SPEC_COMMIT ($n crud/unified json files)"
|