M3: the implication test, and the row closes #14
33
PLAN.md
33
PLAN.md
@@ -467,7 +467,7 @@ answers, and the trade is only acceptable because the lie is removed first.
|
||||
| M1 | **Cursors + wire polish** | getMore / killCursors / batchSize; server-side cursor state with idle timeout; sessions plumbing (lsid accepted) as drivers send it; hello advertisement updates; **`moreToCome` on requests** (see the bug below); command-monitoring assertions in the spec runner | crud spec suite green; e2e green |
|
||||
| M2 | **The `aggregate` command surface** | `$out` and `$merge` (7 of the 13 failures), and refusing every pipeline construct the engine does not implement instead of answering `0` (amendment A6). The other 6 failures are blocked on M2.5, M4 and M8 — see `docs/M2_DESIGN_REVIEW.md` §7 | `aggregate-*.json`: 0 fail among the 7 reachable cases |
|
||||
| M2.5 | **The aggregation engine** | expression evaluator, per-stage document iterator, the accumulators, `$unwind`; `$lookup`/`$facet` explicitly out of the first cut (amendment A6) | a purpose-built stage corpus, every expectation measured against mongod |
|
||||
| M3 | **Update operators + index types** | `distinct` (**done**); positional paths refused rather than destructive (**done**); `$`/`$[]`/`$[<ident>]` + `arrayFilters` (**done**); $setOnInsert, $addToSet, $mul, $min/$max, $pop, $pullAll, $currentDate + `$push`'s modifiers (**done**); pipeline-style updates (**done**); partial indexes (**done**); hashed indexes (**done**); then the implication test that lets a partial index serve a read | `tests/spec/positional/` 0 fail (51), `tests/spec/operators/` 0 fail (125) and `tests/spec/indexes/` 0 fail (42), all recorded from mongod — **green**; the named gate could not see any of the three, see below; remaining crud coverage; e2e3/e2e4 green |
|
||||
| M3 | **Update operators + index types** | `distinct` (**done**); positional paths refused rather than destructive (**done**); `$`/`$[]`/`$[<ident>]` + `arrayFilters` (**done**); $setOnInsert, $addToSet, $mul, $min/$max, $pop, $pullAll, $currentDate + `$push`'s modifiers (**done**); pipeline-style updates (**done**); partial indexes (**done**); hashed indexes (**done**); the implication test that lets a partial index serve a read (**done**) — **the row is closed** | `tests/spec/positional/` 0 fail (51), `tests/spec/operators/` 0 fail (125) and `tests/spec/indexes/` 0 fail (48), all recorded from mongod — **green**; the named gate could not see any of the three, see below; remaining crud coverage; e2e3/e2e4 green |
|
||||
| M4 | **Sessions + transactions** | logical sessions, snapshot isolation on the mmap engine, write concern at commit | sessions + transactions spec suites green |
|
||||
| M5 | **Change streams** | change feed + resume tokens (likely log-seq based), getMore integration | change-streams spec suite green |
|
||||
| M6 | **Admin/ops commands** | dbStats, collStats, serverStatus, ping, buildInfo, listDatabases filters, dropDatabase durability (log it) | mongosh UX smoke; e2e green |
|
||||
@@ -1151,6 +1151,7 @@ has to be its own commit with its own re-recorded scorecard.
|
||||
to read from a partial index**: it holds a subset, so answering a query from
|
||||
it is only correct when the query implies the filter, and that implication
|
||||
test is the last step of the row. Too few documents is worse than no index.
|
||||
(That step has since landed; see the implication entry below.)
|
||||
|
||||
**Hashed indexes landed**; `hashed.json` is 18/18, so `tests/spec/indexes/`
|
||||
is 42/42. A hashed component stores a tag byte and a 64-bit hash of the
|
||||
@@ -1181,7 +1182,8 @@ has to be its own commit with its own re-recorded scorecard.
|
||||
cannot tell apart from a plain subdocument — so the check walks the path
|
||||
rather than counting what it yields.
|
||||
|
||||
**`tests/spec/indexes/` is the gate**, recorded red at 3/39 across 42 cases.
|
||||
**`tests/spec/indexes/` is the gate**, recorded red at 3/39 across 42 cases
|
||||
and since grown to 48 by the implication work.
|
||||
A case there is a *sequence* -- create, insert, read, list -- because an
|
||||
index outlives a `deleteMany` and every case is about which indexes exist.
|
||||
Recording it corrected two of the review's own guesses: `$in` in a partial
|
||||
@@ -1195,6 +1197,33 @@ has to be its own commit with its own re-recorded scorecard.
|
||||
exists the safe rule is to maintain the index and never read from it. Too
|
||||
few documents is the one failure worse than no index at all.
|
||||
|
||||
- **The implication test closes M3's row, and the two gates see opposite
|
||||
halves of it.** A partial index may answer a query that cannot match a
|
||||
document its filter left out. Two routes: the query pins a value at the
|
||||
filter's path and the *real matcher* is run against a stand-in document
|
||||
holding it (sound because every operator the partial grammar admits is
|
||||
existential, so more values at a path can only satisfy it more easily), or
|
||||
the query's bounds are at least as tight as the filter's. An array value and
|
||||
a null value are refused rather than approximated — `{a: [1, 2]}` matches
|
||||
`{a: [[1, 2], 3]}`, whose values at `a` do not include 1 or 2, and
|
||||
`{a: null}` matches a document with no `a` at all, which has no values there
|
||||
rather than more of them.
|
||||
|
||||
Soundness is judged against **this server's** matcher, not mongod's:
|
||||
`query.matches_bytes` decides the index's contents in `build_entries` and
|
||||
re-filters every candidate. Where this server's comparison differs from
|
||||
mongod's — the type-bracketing item below — both halves are wrong together,
|
||||
which is a matching bug and not a lost document.
|
||||
|
||||
Measured with two mutations, and worth keeping because the answer is not
|
||||
symmetric. Forcing the test to `true` takes `partial.json` to 23/7 and every
|
||||
failure reads "expected N, got N-1" — the exact shape of the bug. Forcing it
|
||||
to `false`, which is the behaviour it replaced, leaves the corpus **30/30**:
|
||||
no client can observe *that* an index was used, only that an answer went
|
||||
missing. So the corpus guards soundness and the unit test on `plan()` is the
|
||||
only thing that can see the feature work at all. A read path with no
|
||||
`explain` has no third option.
|
||||
|
||||
- **What the index corpus caught that was not about indexes: `{a: null}` did
|
||||
not match a missing field.** `find({a: null})` has to match a document with
|
||||
no `a` as well as one holding an explicit null. This server matched only the
|
||||
|
||||
@@ -154,7 +154,8 @@ list used for its own format change and it holds here for the same reason.
|
||||
|
||||
## Outcome
|
||||
|
||||
Steps 1–4 landed in that order. `tests/spec/indexes/` is 42/42.
|
||||
All five steps landed in that order, which closes M3's row.
|
||||
`tests/spec/indexes/` is 48/48.
|
||||
|
||||
Three things this review got wrong, kept here because the point of writing it
|
||||
before the code was to find out which parts would not survive contact:
|
||||
@@ -175,8 +176,22 @@ And one thing the corpus found that this review had no reason to look for:
|
||||
`find({a: null})` did not match a document with no `a`, index or no index. See
|
||||
PLAN §6.
|
||||
|
||||
Step 5, the implication test, is still open. Until it exists a partial index
|
||||
is maintained, enforces `unique`, and is never read from.
|
||||
And one thing §4 asserted without saying how: the implication test itself. It
|
||||
turned out not to need a comparison of its own. Every operator
|
||||
`check_partial_filter` admits is *existential* — "some value at this path
|
||||
satisfies it" — so when the query pins a value, running the real matcher
|
||||
against a stand-in document holding it settles `$eq`, `$in`, `$type`,
|
||||
`$exists` and the bounds together. Only two shapes escape that argument (an
|
||||
array value, whose elements a real document need not have; and null, which
|
||||
also matches a document with no such field), and only ranges need reasoning
|
||||
of their own.
|
||||
|
||||
The unnoticed half was the gates. A corpus recorded from mongod can see an
|
||||
implication test that says *yes* too readily — the answer loses documents —
|
||||
but not one that never says yes at all, because no client can observe which
|
||||
index a read used. Measured: forcing the test true takes `partial.json` to
|
||||
23/7; forcing it false leaves it 30/30. The unit test on `plan()` is the only
|
||||
thing that sees the feature work.
|
||||
|
||||
## 6. Not covered
|
||||
|
||||
|
||||
@@ -50,7 +50,7 @@ halves have since been driven green:
|
||||
|
||||
```
|
||||
hashed.json 18 pass 0 fail 0 skip
|
||||
partial.json 24 pass 0 fail 0 skip
|
||||
partial.json 30 pass 0 fail 0 skip
|
||||
```
|
||||
|
||||
The three that passed at the start were the reads a partial index does not
|
||||
@@ -63,6 +63,24 @@ to match a document with no `a`, and this server matched only an explicit
|
||||
null — with or without an index. No other test in the repository asks, and
|
||||
the pinned crud+aggregate scorecard did not move when it was fixed.
|
||||
|
||||
## What this corpus cannot see
|
||||
|
||||
Six of `partial.json`'s cases exist for the implication test — the rule that
|
||||
lets a partial index answer a read — and it is worth being exact about what
|
||||
they guard, because it is only half of it.
|
||||
|
||||
A partial index holds a subset, so reading from one the query does not imply
|
||||
returns **too few** documents, and that is an answer a result comparison
|
||||
catches. It was measured: forcing the implication test to always say yes takes
|
||||
this file to 23 pass / 7 fail, every failure reading "expected N, got N-1".
|
||||
|
||||
The other direction is invisible here. Forcing it to always say *no* — the
|
||||
behaviour before the test existed, where a partial index was maintained and
|
||||
never read — leaves this file at **30 pass / 0 fail**. No client can observe
|
||||
which index a read used, and this server has no `explain`. So the corpus
|
||||
guards soundness and a unit test on `plan()` is the only thing that sees the
|
||||
feature work at all.
|
||||
|
||||
## What recording it settled
|
||||
|
||||
Two of the review's own guesses were wrong, which is why it was recorded
|
||||
|
||||
Reference in New Issue
Block a user