All five steps of the index-types review landed:
the refusal, the recorded corpus, partial indexes, hashed indexes, and
the implication test. `tests/spec/indexes/` is 48/48 and all four
recorded corpora are green -- positional 51, operators 125, indexes 48,
aggregate 70.
The review gets a fourth correction, and it is about method rather than a
fact. §4 asserted the implication rule without saying how to test it; the
answer needed no comparison of its own, because every operator the
partial-filter grammar admits is existential, so running the real matcher
against a stand-in document settles five operators at once.
What the review missed entirely was the gates. A corpus recorded from
mongod can see an implication test that says yes too readily -- documents
go missing -- but not one that never says yes, because no client can
observe which index a read used and this server has no `explain`. Both
numbers are recorded in PLAN §6 and the corpus README so the next change
to this code knows which gate is load-bearing for which direction.
Nothing else in M3 remains. PLAN §6 still carries the items this row
turned up and deliberately left: comparison operators are not
type-bracketed, a dotted path through an empty array reads as absent, a
key-pattern direction may be any non-zero number, three positional
divergences, three operator omissions, and the four older engine items.
Step 5, the last of `docs/M3_INDEX_TYPES_DESIGN_REVIEW.md`'s order and the
last item in M3's row. A partial index was maintained and enforced
`unique`, and every read scanned -- correct, but the speedup the option
exists for was never earned.
The test is one-sided by construction: a `false` costs a scan, a `true`
has to be right, because returning too few documents is the one failure
worse than having no index at all. Two routes, and a filter conjunct is
implied if either answers yes:
**Route one, the query pins a value.** Run the *real matcher* against a
stand-in document holding that value at the path, rather than
reimplementing eight operators against a comparison that would then have
two definitions. Sound because every operator `check_partial_filter`
admits is existential -- "some value at this path satisfies it" -- so a
document with more values at the path satisfies it at least as easily,
and every document the query matches has the pinned value among its
values there. Covers `$eq`, `$in`, `$type`, `$exists` and the bounds in
one stroke.
Two shapes break that argument and are refused rather than approximated,
and both have a row in the test table:
- an **array** value. `{a: [1, 2]}` matches `{a: [[1, 2], 3]}`, whose
values at `a` do not include 1 or 2 -- only one level is expanded, so
the real document's value set is not a superset of the stand-in's.
- a **null** value. `{a: null}` also matches a document with no `a`,
which has no values at the path rather than more of them. The
stand-in alone would report `{a: {$exists: true}}` as implied, so an
empty document is tested too and both have to agree. This is the rule
the previous commit's null fix made necessary and possible in the
same breath.
**Route two, bounds.** The only route needing neither side to name a
document: `{a: {$gt: 5}}` implies `{a: {$gt: 0}}`. Inclusivity is where
it is decided -- `$gte: 0` admits the endpoint that `$gt: 0` excludes.
Soundness is judged against *this server's* matcher, not mongod's. Both
halves of the question run the same code: `query.matches_bytes` decides
the index's contents in `build_entries` and re-filters every candidate
the plan yields. Where this server's comparison differs from mongod's
(PLAN §6: the comparison operators are not type-bracketed) both halves
are wrong together, which is a matching bug and not a lost document.
`$or` on the filter's side is implied by one implied branch: sufficient,
not necessary, since a query can imply a disjunction without implying a
disjunct.
**What each gate can and cannot see, measured with two mutations.** With
`query_implies_filter` forced to `true`, `partial.json` goes 23/7 and
every failure reads "expected N, got N-1" -- the exact shape of the bug.
With it forced to `false` -- the behaviour this commit replaces -- the
corpus is 30/30, because no client can observe *that* an index was used,
only that an answer went missing. So the corpus guards soundness and the
unit test on `plan()` is the only thing that sees the feature work at
all; both are needed and the commit says which does which.
Six corpus cases added, each pairing a query that implies the filter with
one that does not and touches the same field: a query leaving the
filter's field out, an `$in` straddling the filter, a query for null
against an `$exists` filter, equalities inside and outside a range
filter, a sort a partial index could serve, and a unique partial index
read. `partial.json` 24 -> 30 cases, `tests/spec/indexes/` 42 -> 48.
Verified: 257/257 unit tests in ReleaseFast and ReleaseSafe, 88/88 fuzz,
all four corpora 0 fail, pinned scorecard unchanged at 228/63/196, the
full e2e matrix and crash-fuzz green.
`tests/spec/indexes/` is 42/42, so all four recorded corpora are green:
positional 51, operators 125, indexes 42, aggregate 70.
Records what implementing the row taught, including the three things the
design review got wrong. Two were already noted when the corpus was
recorded ($in is allowed; a differing filter is 86); the third is new and
cheaper than the review's version: hashed needs no flags bit, because it
belongs to a key *component* and the per-component direction byte was
already there holding 0 or 1.
Also fixes a corpus case that did not measure what it said. "equality
across numeric types" sent an int32, because a source is plain JSON and
`5.0` is `5` after JSON.parse -- it was a second copy of the case above
it. Reading sources as EJSON was tried and reverted, and the recorder now
says why: EJSON's wrapper namespace collides with the query operators
these sources are made of, so `{"$regex": "x"}` became a BSONRegExp,
`structuredClone` flattened it to `{pattern, options}`, and "a filter
using $regex is refused" silently became a filter mongod accepts. The
cross-type property is a unit test instead, which is where it belongs --
it is about how this server hashes, and mongod's hash is a different
function, so a corpus could only ever check the answer.
The case is renamed to what it does measure rather than deleted: two
documents sharing a value is still the read a hashed index exists for.
Verified: 256/256 unit tests in ReleaseFast and ReleaseSafe, 87/87 fuzz,
all four corpora 0 fail, pinned scorecard unchanged at 228/63/196, the
full e2e matrix and crash-fuzz green.
partial.json 3/24 -> 24/24, hashed.json still 0/18. PLAN §6 records the
planner rule that is deliberately left conservative -- a partial index is
maintained and enforces `unique`, and reads scan until the implication test
exists.
Full matrix at this commit: 250/250 unit tests in ReleaseFast and ReleaseSafe,
83/83 fuzz, operators 125/0, positional 51/0, aggregation 70/0, pinned crud
228/63/196, e2e and crash-fuzz green.
M3's last row, and the first of the four corpora here whose subject nothing in
the repository tested at all: the pinned suite is crud and aggregate, and
`e2e5.js`/`e2e6.js` write neither a partial nor a hashed spec.
42 cases in two files, recorded red at 3/39. The three that pass are the reads
a partial index does not change -- this server indexes every document, so a
query still finds everything, which is exactly why the review called this a
smaller fire than `arrayFilters`.
A case here is a *sequence* rather than one operation: create an index, insert
against it, read back, list it. So the recorder walks a case's operations in
order and stops at the first that throws, which is what a client would see,
and drops the collection between cases because an index outlives a
`deleteMany`.
Two of the design review's own guesses were wrong, which is the argument for
recording rather than reasoning:
- **`$in` in a partial filter is allowed.** The review grouped it with `$ne`
and `$regex`, which are 67.
- **Same key, different filter, no explicit name is IndexKeySpecsConflict
(86)**, not the 67 the review assumed.
What it confirmed, and what the implementation now has to satisfy: a unique
partial index constrains only the documents its filter selects; a document
*leaving* the filter frees the value it held; a document *entering* it must
take a value nothing inside holds, or the update is E11000. `sparse` and
`partialFilterExpression` may not be combined (67). `expireAfterSeconds` and a
filter may, and `listIndexes` reports the filter before the expiry. Two hashed
components is 31303, `unique` on a hashed index is 16764, and an array at a
hashed path is 16766 *at insert time* rather than at creation.
