tests/spec: MongoDB spec-test runner and the M0 scorecard

PLAN D2 makes the official specification suites the gate for command semantics;
D7.6 asks for the harness to exist at M0 with a recorded baseline. This is that
harness, pinned on both sides -- mongodb/specifications @ 615e0f9 and
mongodb@7.5.0 -- because a scorecard is only comparable across milestones if a
delta cannot be an upstream test change.

It implements the unified format's Evaluating Matches algorithm as written,
including the two rules that decide whether a pass is earned: extra keys are
tolerated only in a root document, and numeric types compare flexibly. Anything
unimplemented is a SKIP with a reason, never a pass, and the one assertion class
not yet checked -- expectEvents, i.e. command monitoring -- is disclosed at the
top of the scorecard so `pass` reads as an upper bound.

First honest run: 131 pass, 161 fail, 195 skip over 175 files, zero timeouts.

Getting there took four attempts, and the failures are documented in the README
because each would have shipped a scorecard claiming a compatibility gap that
did not exist. Two were genuine leaks in this runner (clients left open when a
case timed out; clients registered for cleanup only after `await connect()`,
plus abandoned cases still creating more). The third I misdiagnosed as machine
load. The fourth attempt found the real cause: a leaked catalog lock in the
engine, fixed separately, which alone accounts for the jump from 45 passes to
131.

So the runner carries its own guards: per-operation CSOT timeouts so work is
never abandoned, an active-handle census per file, an end-of-run tripwire for
stray timers, a hard stop if the server dies rather than emitting hundreds of
misleading ECONNREFUSED failures, and --skip/--limit for bisecting a run whose
failures depend on position. The README states the rule plainly -- a long
unbroken tail of timeouts is a harness bug until proven otherwise -- and the two
commands that settle it.

Also fixes bench-run.sh, which copied its report over bench-latest.txt
unconditionally, including after a run that only warned -- so a degraded run
could silently replace the baseline that PLAN D7.5 makes a milestone gate.
This commit is contained in:
2026-08-03 18:58:01 +03:00
parent e2c25a986b
commit 90de7820da
7 changed files with 1487 additions and 6 deletions

52
PLAN.md
View File

@@ -380,11 +380,36 @@ stay in the arena — it outlives the command; it is only the ArrayList's own
buffer whose allocator has to match its `deinit`). Add an e2e case for a bare
`$sort` pipeline, and mutation-check it by restoring `arena` on the append.
### Bug found by the spec harness: a nameless command leaked the catalog lock
**Fixed. Severity: permanent denial of service, remotely triggerable by an
ordinary query.** This one blocked the M0 scorecard outright and cost three
invalid baselines.
`dispatch` resolved the namespace *after* taking the catalog lock and bailed
with `orelse return` when the collection name was missing. A plain return runs
neither the `errdefer` nor the explicit unlocks, so the lock was held — shared —
for the life of the process. `db.aggregate(...)` reaches it: that sends
`{aggregate: 1}`, whose value is not a string.
Worth understanding how it hid, because the shape recurs: a leaked *shared* lock
is invisible to readers. `ping` and `listDatabases` kept answering in
microseconds and an external prober saw `ok 15ms` straight through the hang, so
the server looked healthy. Only a write needing the catalog exclusive to create
a collection blocked — so the damage appeared one command later, on a different
connection, as a client-side timeout with nothing linking it to the cause.
What actually found it: the driver's own command log, showing an insert sitting
for exactly `socketTimeoutMS` against an idle engine.
Namespace resolution now happens before any lock is taken. Two lessons kept in
`tests/spec/README.md`: a responsive server does not exonerate the engine, and
probe with the operation that is stuck rather than with `ping`.
### Bug found by the spec harness: unacknowledged writes corrupt the connection
Recorded here rather than fixed in M0, since it is M1's surface — but it is a
correctness bug, not a missing feature, and it is worth doing early because it
is a handful of lines.
**Fixed** (in M0 rather than M1 as originally recorded — it is a correctness
bug, not a missing feature, and it is a handful of lines).
`wire.Message.flags` is parsed and stored but **never read**. A driver sending
an unacknowledged write (`writeConcern: {w: 0}`) sets `moreToCome` (bit 0x2)
@@ -403,9 +428,24 @@ next command on same connection FAILED: MongoUnexpectedServerResponseError:
performance choice — breaks a connection on first use, and it is invisible to
the existing e2e suites because none of them use it.
Fix: when `flags & 0x2` is set on an OP_MSG request, run the command and write
no reply. Add an e2e case for it (unacknowledged write, then a read on the same
connection), and mutation-check it by clearing the flag test.
Fixed by suppressing the reply when `flags & 0x2` is set on an OP_MSG request:
the command still runs. The e2e case pins `maxPoolSize` to 1, since with a
larger pool the driver may hand the next operation a different connection and
hide it.
### What the harness was worth
Its first honest run — after the two bugs above and two genuine leaks in the
runner itself — reports **131 pass, 161 fail, 195 skip** over 175 files, with
zero timeouts. Before the catalog-lock fix the same suite reported 45 passes:
that gap is the measure of what one leaked lock was hiding, and of why a
scorecard is only worth committing once it disagrees with nothing that passes in
isolation.
The remaining failures are real work, and they cluster usefully: update-operator
gaps (`bad update`, `update must be a document` — M3), unimplemented commands
(`distinct`, `$merge`, `$out` — M2/M3), and result-shape mismatches in
`bulkWrite`/`insertMany`. That list, not the total, is the milestone backlog.
---