tests/spec: MongoDB spec-test runner and the M0 scorecard
PLAN D2 makes the official specification suites the gate for command semantics; D7.6 asks for the harness to exist at M0 with a recorded baseline. This is that harness, pinned on both sides -- mongodb/specifications @ 615e0f9 and mongodb@7.5.0 -- because a scorecard is only comparable across milestones if a delta cannot be an upstream test change. It implements the unified format's Evaluating Matches algorithm as written, including the two rules that decide whether a pass is earned: extra keys are tolerated only in a root document, and numeric types compare flexibly. Anything unimplemented is a SKIP with a reason, never a pass, and the one assertion class not yet checked -- expectEvents, i.e. command monitoring -- is disclosed at the top of the scorecard so `pass` reads as an upper bound. First honest run: 131 pass, 161 fail, 195 skip over 175 files, zero timeouts. Getting there took four attempts, and the failures are documented in the README because each would have shipped a scorecard claiming a compatibility gap that did not exist. Two were genuine leaks in this runner (clients left open when a case timed out; clients registered for cleanup only after `await connect()`, plus abandoned cases still creating more). The third I misdiagnosed as machine load. The fourth attempt found the real cause: a leaked catalog lock in the engine, fixed separately, which alone accounts for the jump from 45 passes to 131. So the runner carries its own guards: per-operation CSOT timeouts so work is never abandoned, an active-handle census per file, an end-of-run tripwire for stray timers, a hard stop if the server dies rather than emitting hundreds of misleading ECONNREFUSED failures, and --skip/--limit for bisecting a run whose failures depend on position. The README states the rule plainly -- a long unbroken tail of timeouts is a harness bug until proven otherwise -- and the two commands that settle it. Also fixes bench-run.sh, which copied its report over bench-latest.txt unconditionally, including after a run that only warned -- so a degraded run could silently replace the baseline that PLAN D7.5 makes a milestone gate.
This commit is contained in:
52
PLAN.md
52
PLAN.md
@@ -380,11 +380,36 @@ stay in the arena — it outlives the command; it is only the ArrayList's own
|
||||
buffer whose allocator has to match its `deinit`). Add an e2e case for a bare
|
||||
`$sort` pipeline, and mutation-check it by restoring `arena` on the append.
|
||||
|
||||
### Bug found by the spec harness: a nameless command leaked the catalog lock
|
||||
|
||||
**Fixed. Severity: permanent denial of service, remotely triggerable by an
|
||||
ordinary query.** This one blocked the M0 scorecard outright and cost three
|
||||
invalid baselines.
|
||||
|
||||
`dispatch` resolved the namespace *after* taking the catalog lock and bailed
|
||||
with `orelse return` when the collection name was missing. A plain return runs
|
||||
neither the `errdefer` nor the explicit unlocks, so the lock was held — shared —
|
||||
for the life of the process. `db.aggregate(...)` reaches it: that sends
|
||||
`{aggregate: 1}`, whose value is not a string.
|
||||
|
||||
Worth understanding how it hid, because the shape recurs: a leaked *shared* lock
|
||||
is invisible to readers. `ping` and `listDatabases` kept answering in
|
||||
microseconds and an external prober saw `ok 15ms` straight through the hang, so
|
||||
the server looked healthy. Only a write needing the catalog exclusive to create
|
||||
a collection blocked — so the damage appeared one command later, on a different
|
||||
connection, as a client-side timeout with nothing linking it to the cause.
|
||||
|
||||
What actually found it: the driver's own command log, showing an insert sitting
|
||||
for exactly `socketTimeoutMS` against an idle engine.
|
||||
|
||||
Namespace resolution now happens before any lock is taken. Two lessons kept in
|
||||
`tests/spec/README.md`: a responsive server does not exonerate the engine, and
|
||||
probe with the operation that is stuck rather than with `ping`.
|
||||
|
||||
### Bug found by the spec harness: unacknowledged writes corrupt the connection
|
||||
|
||||
Recorded here rather than fixed in M0, since it is M1's surface — but it is a
|
||||
correctness bug, not a missing feature, and it is worth doing early because it
|
||||
is a handful of lines.
|
||||
**Fixed** (in M0 rather than M1 as originally recorded — it is a correctness
|
||||
bug, not a missing feature, and it is a handful of lines).
|
||||
|
||||
`wire.Message.flags` is parsed and stored but **never read**. A driver sending
|
||||
an unacknowledged write (`writeConcern: {w: 0}`) sets `moreToCome` (bit 0x2)
|
||||
@@ -403,9 +428,24 @@ next command on same connection FAILED: MongoUnexpectedServerResponseError:
|
||||
performance choice — breaks a connection on first use, and it is invisible to
|
||||
the existing e2e suites because none of them use it.
|
||||
|
||||
Fix: when `flags & 0x2` is set on an OP_MSG request, run the command and write
|
||||
no reply. Add an e2e case for it (unacknowledged write, then a read on the same
|
||||
connection), and mutation-check it by clearing the flag test.
|
||||
Fixed by suppressing the reply when `flags & 0x2` is set on an OP_MSG request:
|
||||
the command still runs. The e2e case pins `maxPoolSize` to 1, since with a
|
||||
larger pool the driver may hand the next operation a different connection and
|
||||
hide it.
|
||||
|
||||
### What the harness was worth
|
||||
|
||||
Its first honest run — after the two bugs above and two genuine leaks in the
|
||||
runner itself — reports **131 pass, 161 fail, 195 skip** over 175 files, with
|
||||
zero timeouts. Before the catalog-lock fix the same suite reported 45 passes:
|
||||
that gap is the measure of what one leaked lock was hiding, and of why a
|
||||
scorecard is only worth committing once it disagrees with nothing that passes in
|
||||
isolation.
|
||||
|
||||
The remaining failures are real work, and they cluster usefully: update-operator
|
||||
gaps (`bad update`, `update must be a document` — M3), unimplemented commands
|
||||
(`distinct`, `$merge`, `$out` — M2/M3), and result-shape mismatches in
|
||||
`bulkWrite`/`insertMany`. That list, not the total, is the milestone backlog.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user