7de3e6b666
Step 5, the last of `docs/M3_INDEX_TYPES_DESIGN_REVIEW.md`'s order and the
last item in M3's row. A partial index was maintained and enforced
`unique`, and every read scanned -- correct, but the speedup the option
exists for was never earned.
The test is one-sided by construction: a `false` costs a scan, a `true`
has to be right, because returning too few documents is the one failure
worse than having no index at all. Two routes, and a filter conjunct is
implied if either answers yes:
**Route one, the query pins a value.** Run the *real matcher* against a
stand-in document holding that value at the path, rather than
reimplementing eight operators against a comparison that would then have
two definitions. Sound because every operator `check_partial_filter`
admits is existential -- "some value at this path satisfies it" -- so a
document with more values at the path satisfies it at least as easily,
and every document the query matches has the pinned value among its
values there. Covers `$eq`, `$in`, `$type`, `$exists` and the bounds in
one stroke.
Two shapes break that argument and are refused rather than approximated,
and both have a row in the test table:
- an **array** value. `{a: [1, 2]}` matches `{a: [[1, 2], 3]}`, whose
values at `a` do not include 1 or 2 -- only one level is expanded, so
the real document's value set is not a superset of the stand-in's.
- a **null** value. `{a: null}` also matches a document with no `a`,
which has no values at the path rather than more of them. The
stand-in alone would report `{a: {$exists: true}}` as implied, so an
empty document is tested too and both have to agree. This is the rule
the previous commit's null fix made necessary and possible in the
same breath.
**Route two, bounds.** The only route needing neither side to name a
document: `{a: {$gt: 5}}` implies `{a: {$gt: 0}}`. Inclusivity is where
it is decided -- `$gte: 0` admits the endpoint that `$gt: 0` excludes.
Soundness is judged against *this server's* matcher, not mongod's. Both
halves of the question run the same code: `query.matches_bytes` decides
the index's contents in `build_entries` and re-filters every candidate
the plan yields. Where this server's comparison differs from mongod's
(PLAN §6: the comparison operators are not type-bracketed) both halves
are wrong together, which is a matching bug and not a lost document.
`$or` on the filter's side is implied by one implied branch: sufficient,
not necessary, since a query can imply a disjunction without implying a
disjunct.
**What each gate can and cannot see, measured with two mutations.** With
`query_implies_filter` forced to `true`, `partial.json` goes 23/7 and
every failure reads "expected N, got N-1" -- the exact shape of the bug.
With it forced to `false` -- the behaviour this commit replaces -- the
corpus is 30/30, because no client can observe *that* an index was used,
only that an answer went missing. So the corpus guards soundness and the
unit test on `plan()` is the only thing that sees the feature work at
all; both are needed and the commit says which does which.
Six corpus cases added, each pairing a query that implies the filter with
one that does not and touches the same field: a query leaving the
filter's field out, an `$in` straddling the filter, a query for null
against an `$exists` filter, equalities inside and outside a range
filter, a sort a partial index could serve, and a unique partial index
read. `partial.json` 24 -> 30 cases, `tests/spec/indexes/` 42 -> 48.
Verified: 257/257 unit tests in ReleaseFast and ReleaseSafe, 88/88 fuzz,
all four corpora 0 fail, pinned scorecard unchanged at 228/63/196, the
full e2e matrix and crash-fuzz green.
The index corpus
M3's last row: partial and hashed indexes. docs/M3_INDEX_TYPES_DESIGN_REVIEW.md
measured both against mongod 8.3.7 and found they were not the same kind of
gap — hashed was honestly refused, partialFilterExpression was accepted and
ignored, and a unique partial index therefore refused inserts mongod
accepts.
It also found that no test in this repository covered that row, in any
suite. The pinned corpus is crud and aggregate; tests/e2e/e2e5.js and
e2e6.js test indexes and write neither a partial nor a hashed spec. So this
directory exists for the same reason tests/spec/positional/ and
tests/spec/operators/ do.
The one rule
Inputs are authored here; expectations are measured against a real mongod.
tests/spec/indexes/
sources/*.json documents + operations, authored
record.js runs them against mongod, writes the expectations
*.json generated, unified format, do not hand-edit
mongod --port 27099 --dbpath <dir>
node tests/spec/indexes/record.js --mongod-port 27099
node tests/spec/run.js --suite-dir tests/spec/indexes
Unlike the other two corpora, a case here is a sequence: create an index,
insert against it, read back, list it. An index outlives a deleteMany, and
every case is about which indexes exist, so the recorder drops the collection
between cases and walks each case's operations in order — stopping at the
first that throws, which is what a client would see.
Sources are plain JSON, not EJSON, and that costs something worth stating: a
source cannot name a BSON type the JSON grammar has no syntax for, so 5.0
reaches the driver as an int32. Reading them as EJSON was tried and reverted —
EJSON's wrapper namespace collides with the query operators these sources are
made of. {"$regex": "x"} parses to a BSONRegExp, structuredClone
flattens it to {pattern, options}, and "a filter using $regex is refused"
silently became a filter mongod accepts.
Where it stands
Recorded against mongod 8.3.7 at 3/39 -- red by construction -- and both halves have since been driven green:
hashed.json 18 pass 0 fail 0 skip
partial.json 24 pass 0 fail 0 skip
The three that passed at the start were the reads a partial index does not
change: this server indexed every document, so a query still found
everything, which is the whole reason the review called the partial gap
smaller than the arrayFilters one.
The last one to go green was not about indexes at all. find({a: null}) has
to match a document with no a, and this server matched only an explicit
null — with or without an index. No other test in the repository asks, and
the pinned crud+aggregate scorecard did not move when it was fixed.
What recording it settled
Two of the review's own guesses were wrong, which is why it was recorded rather than reasoned:
| mongod | |
|---|---|
$in in a partial filter |
allowed — the review listed it with $ne and $regex |
| same key, different filter, no explicit name | IndexKeySpecsConflict (86), not the 67 the review assumed |
And what it confirmed:
| mongod | |
|---|---|
| a unique partial index | constrains only the documents its filter selects — two {a: 1, t: false} are fine, two {a: 9, t: true} are E11000 |
| a document leaving the filter | frees the value it held for another document to take |
| a document entering the filter | must take a value nothing inside it holds, or the update is E11000 |
$regex, $ne in a filter |
67 |
| a filter that is not a document | TypeMismatch (14) |
sparse + partialFilterExpression |
67 — may not be combined |
expireAfterSeconds + a filter |
allowed; listIndexes reports the filter before the expiry |
| an empty filter | allowed, and reported |
| two hashed components | 31303 |
unique on a hashed index |
16764 |
| an array at a hashed path | 16766, at insert time, not at creation |
a key direction that is not 1, -1 or "hashed" |
67 |
| a hashed index beside an ascending one on the same field | both exist |
| a range query or a sort over a hashed field | still correct — the planner declines the index rather than misusing it |
And what implementing hashed then measured, none of which the review had:
| mongod | |
|---|---|
codeName for 31303 and 16764 |
Location31303 / Location16764 — bare location numbers with no name |
codeName for {a: "bogus"} |
CannotCreateIndex, the named 67 |
| 16766 on an insert or update | a per-document writeError beside ok: 1, so the rest of the batch lands |
16766 from createIndexes |
a command error, over data that already holds an array |
| a one-element array through the path | refused too — the case a value count cannot tell apart from a plain subdocument |
| an empty array at the path | refused |
| an array inside a subdocument at the path | fine — only the path itself matters |
hashed with expireAfterSeconds, or with a partial filter |
both allowed |
| a direction that is any non-zero, non-NaN number | allowed, sign taken as the direction, value echoed verbatim — this server takes only ±1 (PLAN §6) |