M3: partial indexes #12
Reference in New Issue
Block a user
Delete Branch "m3-partial-indexes"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Step 3 of
docs/M3_INDEX_TYPES_DESIGN_REVIEW.md:partialFilterExpressionishonoured rather than refused.
tests/spec/indexes/partial.jsontests/spec/indexes/hashed.json83/83 fuzz, operators 125/0, positional 51/0, aggregation 70/0, pinned crud
228/63/196 unmoved, the full e2e matrix and
crash-fuzzgreen.One choke point
The filter is consulted in exactly one place --
build_entries-- which iswhat keeps the insert and the remove path from ever disagreeing about which
documents the index holds. Filtering at each call site instead is how an index
ends up with entries pointing at documents that are gone.
That single point is also why
uniquecomes out right for free, anduniqueis the whole reason this was a wrong answer rather than a missing feature:
two documents sharing a value outside the filter are now accepted, where
before they were E11000. Unique-within-a-subset is what the option is for.
The planner declines to read from it
A partial index holds a subset, so answering a query from it is only correct
when the query's predicates imply its filter — and that implication test is
step 5, not this one. Returning too few documents is the one failure worse
than having no index at all. So the index is maintained, it enforces
unique,and reads scan. Recorded in PLAN §6 rather than left implicit.
Persistence
A fifth bit in
write_index_catalog's flags byte and the serialized filterafter the TTL. A file written before this never sets the bit and reads back
exactly as it did, so
catalog_versionstays 1 — the same argument the freelist used for its own format change. The reopen half has its own test, with
the mutation named: drop the flag bit and the reopened index covers every
document and starts refusing the writes the first half just proved legal.
Measured, not assumed
$eq,$gt,$gte,$lt,$lte,$in,$exists,$type,$and,$or;$neand$regex— including a regex sent as a BSON value,which is how a driver spells
{$regex: "x"}and which only the corpuscaught;
sparse+partialFilterExpressionmay not be combined: a sparse index isa partial one whose filter is
{<path>: {$exists: true}}, and a documentsatisfying one and not the other has no defined answer;
option is IndexOptionsConflict (85) next door. The split is that a filter
decides which documents the index is over rather than how it behaves.
ReleaseSafe earned its keep
An assertion held that a non-sparse index covers every document after an open.
A partial one is a third shape that does not — multikey was already the second
— so the assertion is extended rather than relaxed. It fired on the first
ReleaseSafe run of the new test and on no ReleaseFast run, which is the trap
AGENTS.mdwarns about, working.And one runner fix, in its own commit
namesits in the runner'sPOSITIONALset becausedropIndextakes it thatway.
createIndexdoes not — it is an option there, andoptions()wasstripping it, so a corpus case asking for a named index silently got a derived
one. Two
partial.jsoncases failed on exactly that.nameis an option of createIndex, not a positional dc26c66f40`partialFilterExpression` is honoured rather than refused. The filter is consulted in exactly one place -- `build_entries` -- which is what keeps the insert and the remove path from ever disagreeing about which documents the index holds. Filtering at each call site instead is how an index ends up with entries pointing at documents that are gone. That single choke point is also why `unique` comes out right for free, and `unique` is the whole reason this was a wrong answer rather than a missing feature: two documents sharing a value *outside* the filter are now accepted, where before they were E11000. Unique-within-a-subset is what the option is for. The filter is part of the index, so it is persisted with it: a fifth bit in `write_index_catalog`'s flags byte and the serialized document after the TTL. A file written before this never sets the bit and reads back exactly as it did, so `catalog_version` stays 1 -- the same argument the free list used. **The planner declines to read from a partial index.** It holds a subset, so answering a query from it is only correct when the query's predicates imply its filter, and that implication test does not exist yet. Returning too few documents is the one failure worse than having no index at all. So it is maintained, it enforces `unique`, and reads scan. PLAN §6. Which predicates a filter may hold is measured: `$eq`, `$gt`, `$gte`, `$lt`, `$lte`, `$in`, `$exists`, `$type`, `$and`, `$or`. `$ne` and `$regex` are refused -- including a regex sent as a BSON *value*, which is how a driver spells `{$regex: "x"}` and which the corpus caught. `sparse` and `partialFilterExpression` may not be combined: a sparse index is a partial one whose filter is `{<path>: {$exists: true}}`, and a document satisfying one and not the other has no defined answer. Same name, different filter is IndexKeySpecsConflict (86) where a differing *option* is IndexOptionsConflict (85) next door -- measured, and the split is that a filter decides which documents the index is over rather than how it behaves. ReleaseSafe earned its keep: an assertion held that a non-sparse index covers every document after an open, and a partial one is a third shape that does not. Extended rather than relaxed -- multikey was already the second. partial.json 3/24 -> **24/24**. 250/250 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz, everything else unmoved.