7de3e6b666
Step 5, the last of `docs/M3_INDEX_TYPES_DESIGN_REVIEW.md`'s order and the
last item in M3's row. A partial index was maintained and enforced
`unique`, and every read scanned -- correct, but the speedup the option
exists for was never earned.
The test is one-sided by construction: a `false` costs a scan, a `true`
has to be right, because returning too few documents is the one failure
worse than having no index at all. Two routes, and a filter conjunct is
implied if either answers yes:
**Route one, the query pins a value.** Run the *real matcher* against a
stand-in document holding that value at the path, rather than
reimplementing eight operators against a comparison that would then have
two definitions. Sound because every operator `check_partial_filter`
admits is existential -- "some value at this path satisfies it" -- so a
document with more values at the path satisfies it at least as easily,
and every document the query matches has the pinned value among its
values there. Covers `$eq`, `$in`, `$type`, `$exists` and the bounds in
one stroke.
Two shapes break that argument and are refused rather than approximated,
and both have a row in the test table:
- an **array** value. `{a: [1, 2]}` matches `{a: [[1, 2], 3]}`, whose
values at `a` do not include 1 or 2 -- only one level is expanded, so
the real document's value set is not a superset of the stand-in's.
- a **null** value. `{a: null}` also matches a document with no `a`,
which has no values at the path rather than more of them. The
stand-in alone would report `{a: {$exists: true}}` as implied, so an
empty document is tested too and both have to agree. This is the rule
the previous commit's null fix made necessary and possible in the
same breath.
**Route two, bounds.** The only route needing neither side to name a
document: `{a: {$gt: 5}}` implies `{a: {$gt: 0}}`. Inclusivity is where
it is decided -- `$gte: 0` admits the endpoint that `$gt: 0` excludes.
Soundness is judged against *this server's* matcher, not mongod's. Both
halves of the question run the same code: `query.matches_bytes` decides
the index's contents in `build_entries` and re-filters every candidate
the plan yields. Where this server's comparison differs from mongod's
(PLAN §6: the comparison operators are not type-bracketed) both halves
are wrong together, which is a matching bug and not a lost document.
`$or` on the filter's side is implied by one implied branch: sufficient,
not necessary, since a query can imply a disjunction without implying a
disjunct.
**What each gate can and cannot see, measured with two mutations.** With
`query_implies_filter` forced to `true`, `partial.json` goes 23/7 and
every failure reads "expected N, got N-1" -- the exact shape of the bug.
With it forced to `false` -- the behaviour this commit replaces -- the
corpus is 30/30, because no client can observe *that* an index was used,
only that an answer went missing. So the corpus guards soundness and the
unit test on `plan()` is the only thing that sees the feature work at
all; both are needed and the commit says which does which.
Six corpus cases added, each pairing a query that implies the filter with
one that does not and touches the same field: a query leaving the
filter's field out, an `$in` straddling the filter, a query for null
against an `$exists` filter, equalities inside and outside a range
filter, a sort a partial index could serve, and a unique partial index
read. `partial.json` 24 -> 30 cases, `tests/spec/indexes/` 42 -> 48.
Verified: 257/257 unit tests in ReleaseFast and ReleaseSafe, 88/88 fuzz,
all four corpora 0 fail, pinned scorecard unchanged at 228/63/196, the
full e2e matrix and crash-fuzz green.
273 lines
12 KiB
JSON
273 lines
12 KiB
JSON
{
|
|
"_comment": [
|
|
"Inputs only. Expectations are measured -- see record.js.",
|
|
"`partialFilterExpression`: which filters may be written, what the index",
|
|
"then reports about itself, and the one behaviour that made ignoring the",
|
|
"option a wrong answer rather than a missing feature -- a unique partial",
|
|
"index constrains only the documents its filter selects."
|
|
],
|
|
"documents": [
|
|
{ "_id": 1, "a": 1, "s": "x" },
|
|
{ "_id": 2, "a": 5, "s": "y" },
|
|
{ "_id": 3, "s": "z" },
|
|
{ "_id": 4, "a": 5, "s": "x" }
|
|
],
|
|
"cases": [
|
|
{
|
|
"description": "a partial index is created and reports its filter",
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "partialFilterExpression": { "a": { "$gte": 5 } } } },
|
|
{ "name": "listIndexes", "arguments": {} }
|
|
]
|
|
},
|
|
{
|
|
"description": "a query the filter covers still answers every match",
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "partialFilterExpression": { "a": { "$gte": 5 } } } },
|
|
{ "name": "find", "arguments": { "filter": { "a": { "$gte": 5 } }, "sort": { "_id": 1 } } }
|
|
]
|
|
},
|
|
{
|
|
"description": "a query the filter excludes still answers every match",
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "partialFilterExpression": { "a": { "$gte": 5 } } } },
|
|
{ "name": "find", "arguments": { "filter": { "a": 1 }, "sort": { "_id": 1 } } }
|
|
]
|
|
},
|
|
{
|
|
"description": "a query straddling the filter still answers every match",
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "partialFilterExpression": { "a": { "$gte": 5 } } } },
|
|
{ "name": "find", "arguments": { "filter": { "a": { "$gte": 1 } }, "sort": { "_id": 1 } } },
|
|
{ "name": "countDocuments", "arguments": { "filter": {} } }
|
|
]
|
|
},
|
|
{
|
|
"description": "a unique partial index accepts duplicates its filter excludes",
|
|
"documents": [],
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "unique": true, "partialFilterExpression": { "t": true } } },
|
|
{ "name": "insertMany", "arguments": { "documents": [{ "_id": 1, "a": 1, "t": false }, { "_id": 2, "a": 1, "t": false }] } },
|
|
{ "name": "find", "arguments": { "filter": {}, "sort": { "_id": 1 } } }
|
|
]
|
|
},
|
|
{
|
|
"description": "a unique partial index refuses duplicates its filter selects",
|
|
"documents": [],
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "unique": true, "partialFilterExpression": { "t": true } } },
|
|
{ "name": "insertMany", "arguments": { "documents": [{ "_id": 1, "a": 9, "t": true }, { "_id": 2, "a": 9, "t": true }] } }
|
|
]
|
|
},
|
|
{
|
|
"description": "a document leaving the filter frees the value it held",
|
|
"documents": [],
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "unique": true, "partialFilterExpression": { "t": true } } },
|
|
{ "name": "insertMany", "arguments": { "documents": [{ "_id": 1, "a": 9, "t": true }] } },
|
|
{ "name": "updateOne", "arguments": { "filter": { "_id": 1 }, "update": { "$set": { "t": false } } } },
|
|
{ "name": "insertOne", "arguments": { "document": { "_id": 2, "a": 9, "t": true } } },
|
|
{ "name": "find", "arguments": { "filter": {}, "sort": { "_id": 1 } } }
|
|
]
|
|
},
|
|
{
|
|
"description": "a document entering the filter takes the value it names",
|
|
"documents": [],
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "unique": true, "partialFilterExpression": { "t": true } } },
|
|
{ "name": "insertMany", "arguments": { "documents": [{ "_id": 1, "a": 9, "t": true }, { "_id": 2, "a": 9, "t": false }] } },
|
|
{ "name": "updateOne", "arguments": { "filter": { "_id": 2 }, "update": { "$set": { "t": true } } } }
|
|
]
|
|
},
|
|
{
|
|
"_comment": [
|
|
"The reads below exist for the implication test: a partial index may",
|
|
"only answer a query that cannot match a document its filter left",
|
|
"out. Each pairs a query that implies the filter with one that does",
|
|
"not and touches the same field, so an implication test that says yes",
|
|
"too readily loses the documents outside the filter -- which is a",
|
|
"wrong answer a result comparison can see, unlike the index being",
|
|
"used at all, which no client can observe."
|
|
],
|
|
"description": "a query that leaves the filter's field out still sees past it",
|
|
"documents": [
|
|
{ "_id": 1, "a": 1, "t": true },
|
|
{ "_id": 2, "a": 1, "t": false },
|
|
{ "_id": 3, "a": 2, "t": true }
|
|
],
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "partialFilterExpression": { "t": true } } },
|
|
{ "name": "find", "arguments": { "filter": { "a": 1 }, "sort": { "_id": 1 } } },
|
|
{ "name": "find", "arguments": { "filter": { "a": 1, "t": true }, "sort": { "_id": 1 } } },
|
|
{ "name": "find", "arguments": { "filter": { "a": 1, "t": false }, "sort": { "_id": 1 } } }
|
|
]
|
|
},
|
|
{
|
|
"description": "an $in straddling the filter still answers every match",
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "partialFilterExpression": { "a": { "$gte": 5 } } } },
|
|
{ "name": "find", "arguments": { "filter": { "a": { "$in": [1, 5] } }, "sort": { "_id": 1 } } },
|
|
{ "name": "find", "arguments": { "filter": { "a": { "$in": [5, 9] } }, "sort": { "_id": 1 } } }
|
|
]
|
|
},
|
|
{
|
|
"description": "a query for null does not belong to an index that requires the field",
|
|
"documents": [
|
|
{ "_id": 1, "a": 1 },
|
|
{ "_id": 2 },
|
|
{ "_id": 3, "a": null }
|
|
],
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "partialFilterExpression": { "a": { "$exists": true } } } },
|
|
{ "name": "find", "arguments": { "filter": { "a": null }, "sort": { "_id": 1 } } },
|
|
{ "name": "find", "arguments": { "filter": { "a": 1 }, "sort": { "_id": 1 } } }
|
|
]
|
|
},
|
|
{
|
|
"description": "an equality inside the filter's range, and one outside it",
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "partialFilterExpression": { "a": { "$gt": 3 } } } },
|
|
{ "name": "find", "arguments": { "filter": { "a": 5 }, "sort": { "_id": 1 } } },
|
|
{ "name": "find", "arguments": { "filter": { "a": 1 }, "sort": { "_id": 1 } } },
|
|
{ "name": "find", "arguments": { "filter": { "a": { "$gt": 4 } }, "sort": { "_id": 1 } } },
|
|
{ "name": "find", "arguments": { "filter": { "a": { "$gt": 0 } }, "sort": { "_id": 1 } } }
|
|
]
|
|
},
|
|
{
|
|
"description": "a sort a partial index could serve",
|
|
"documents": [
|
|
{ "_id": 1, "a": 3, "t": true },
|
|
{ "_id": 2, "a": 1, "t": false },
|
|
{ "_id": 3, "a": 2, "t": true }
|
|
],
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "partialFilterExpression": { "t": true } } },
|
|
{ "name": "find", "arguments": { "filter": { "t": true }, "sort": { "a": 1 } } },
|
|
{ "name": "find", "arguments": { "filter": {}, "sort": { "a": 1 } } }
|
|
]
|
|
},
|
|
{
|
|
"description": "a unique partial index answers the read it constrains",
|
|
"documents": [
|
|
{ "_id": 1, "a": 9, "t": true },
|
|
{ "_id": 2, "a": 9, "t": false }
|
|
],
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "unique": true, "partialFilterExpression": { "t": true } } },
|
|
{ "name": "find", "arguments": { "filter": { "a": 9, "t": true }, "sort": { "_id": 1 } } },
|
|
{ "name": "find", "arguments": { "filter": { "a": 9 }, "sort": { "_id": 1 } } }
|
|
]
|
|
},
|
|
{
|
|
"description": "a filter on $exists",
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "partialFilterExpression": { "a": { "$exists": true } } } },
|
|
{ "name": "listIndexes", "arguments": {} }
|
|
]
|
|
},
|
|
{
|
|
"description": "a filter with two predicates",
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "partialFilterExpression": { "a": { "$gt": 0 }, "s": "x" } } },
|
|
{ "name": "listIndexes", "arguments": {} }
|
|
]
|
|
},
|
|
{
|
|
"description": "a filter with an explicit $and",
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "partialFilterExpression": { "$and": [{ "a": { "$gt": 0 } }] } } },
|
|
{ "name": "listIndexes", "arguments": {} }
|
|
]
|
|
},
|
|
{
|
|
"description": "a filter on a field the index does not name",
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "partialFilterExpression": { "s": "x" } } },
|
|
{ "name": "listIndexes", "arguments": {} }
|
|
]
|
|
},
|
|
{
|
|
"description": "a filter using $regex is refused",
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "partialFilterExpression": { "s": { "$regex": "x" } } } }
|
|
]
|
|
},
|
|
{
|
|
"description": "a filter using $ne is refused",
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "partialFilterExpression": { "a": { "$ne": 1 } } } }
|
|
]
|
|
},
|
|
{
|
|
"description": "a filter using $in is refused",
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "partialFilterExpression": { "a": { "$in": [1, 2] } } } }
|
|
]
|
|
},
|
|
{
|
|
"description": "a filter that is not a document is refused",
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "partialFilterExpression": 1 } }
|
|
]
|
|
},
|
|
{
|
|
"description": "an empty filter",
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "partialFilterExpression": {} } },
|
|
{ "name": "listIndexes", "arguments": {} }
|
|
]
|
|
},
|
|
{
|
|
"description": "sparse and a partial filter may not be combined",
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "sparse": true, "partialFilterExpression": { "a": { "$gt": 0 } } } }
|
|
]
|
|
},
|
|
{
|
|
"description": "a partial filter with a TTL",
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "expireAfterSeconds": 100, "partialFilterExpression": { "a": { "$gt": 0 } } } },
|
|
{ "name": "listIndexes", "arguments": {} }
|
|
]
|
|
},
|
|
{
|
|
"description": "a partial index on a compound key",
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1, "s": 1 }, "partialFilterExpression": { "a": { "$gt": 0 } } } },
|
|
{ "name": "listIndexes", "arguments": {} }
|
|
]
|
|
},
|
|
{
|
|
"description": "two indexes on one key differing only in their filter",
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "name": "one", "partialFilterExpression": { "a": { "$gt": 0 } } } },
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "name": "two", "partialFilterExpression": { "a": { "$gt": 4 } } } },
|
|
{ "name": "listIndexes", "arguments": {} }
|
|
]
|
|
},
|
|
{
|
|
"description": "the same index twice is idempotent",
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "partialFilterExpression": { "a": { "$gt": 0 } } } },
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "partialFilterExpression": { "a": { "$gt": 0 } } } },
|
|
{ "name": "listIndexes", "arguments": {} }
|
|
]
|
|
},
|
|
{
|
|
"description": "the same name with a different filter conflicts",
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "partialFilterExpression": { "a": { "$gt": 0 } } } },
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "partialFilterExpression": { "a": { "$gt": 4 } } } }
|
|
]
|
|
},
|
|
{
|
|
"description": "a partial index is dropped by name",
|
|
"ops": [
|
|
{ "name": "createIndex", "arguments": { "keys": { "a": 1 }, "name": "pa", "partialFilterExpression": { "a": { "$gt": 0 } } } },
|
|
{ "name": "dropIndex", "arguments": { "name": "pa" } },
|
|
{ "name": "listIndexes", "arguments": {} }
|
|
]
|
|
}
|
|
]
|
|
}
|