M3: design review of the index types, and the partialFilterExpression refusal #10

dev · 2026-08-10T19:18:55Z

dev commented

2026-08-10 19:18:55 +00:00

M3's last row, opened the way M2 and arrayFilters were opened: measure
first, write the review, and land the refusal the measurement demands.

The two halves are not the same kind of gap

PLAN §3 names them together. They are not.

Hashed is honestly missing. createIndex({a: "hashed"}) is refused. Wrong
code -- 2 where mongod says 67, and it has four more specific ones besides --
but the right answer. The client is told no, and queries keep working because
there is simply no index.

Partial was accepted and ignored. createIndex reported success, the
index was built over every document rather than the filtered subset, and
listIndexes did not mention the option.

The row that made it a wrong answer rather than a missing feature

An over-inclusive index still answers reads correctly -- it holds a superset,
never a subset -- so this was not the array-destroying class of bug. That is
worth saying plainly. unique is where it stopped being harmless:

createIndex({a: 1}, {unique: true, partialFilterExpression: {t: true}})
insertMany([{a: 1, t: false}, {a: 1, t: false}])
// mongod: accepted -- neither document is in the index, so neither collides
// ours:   E11000 duplicate key error, dup key: {a: 1}

A legal insert refused. Unique-within-a-subset -- unique email among
active accounts, unique external id among synced rows -- is the whole point of
the option, so every use of it hit this.

Nothing in the repository covered that row. The pinned suite is crud and
aggregate; neither e2e5.js nor e2e6.js writes a partial or hashed spec.

What is in this PR

docs/M3_INDEX_TYPES_DESIGN_REVIEW.md -- both features' rules measured
against mongod 8.3.7: which predicates a partial filter may hold, why
sparse and partialFilterExpression may not be combined (67), the four
hashed refusals and the one that fires at insert time rather than at
creation (16766), and the planner rule that decides the whole design --

a partial index may only answer a query whose predicates imply its
filter, so until that test exists the safe rule is to maintain the index
and never read from it. Too few documents is the one failure worse than
no index at all.

Same for hashed, mirrored: equality predicates only, everything else scans.
The hash value itself is a private encoding choice -- nothing a client can
observe depends on it -- which is exactly why sorts may not use the tree.
The refusal. partialFilterExpression is now CannotCreateIndex (67) at
creation, on the same judgement cmd_update already states about an update
spec's sort: "ignoring the field would be the worst of the three possible
answers." The two alternatives were to keep enforcing unique over the
wrong set, or to echo the option back from listIndexes while not
honouring it, which is a larger lie than saying no.

Two rows added to the existing index-spec refusal test, with the mutation
that reddens them written beside them.

Verification

249/249 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz, operators
125/0, positional 51/0, aggregation 70/0, pinned crud 228/63/196 unmoved, the
full e2e matrix including e2e5/e2e6 and crash-fuzz green.

Next, in the review's order

Record tests/spec/indexes/; then partial indexes, maintained and
unique-enforcing with the planner declining to read from them; then hashed,
equality-only; then the implication test that lets a partial index serve a
read. Both need one catalog field, and write_index_catalog's flags byte has
spare bits, so catalog_version stays 1 -- the same argument the free list
used for its own format change.

M3's last row, opened the way M2 and `arrayFilters` were opened: measure first, write the review, and land the refusal the measurement demands. ## The two halves are not the same kind of gap PLAN §3 names them together. They are not. **Hashed is honestly missing.** `createIndex({a: "hashed"})` is refused. Wrong code -- 2 where mongod says 67, and it has four more specific ones besides -- but the right answer. The client is told no, and queries keep working because there is simply no index. **Partial was accepted and ignored.** `createIndex` reported success, the index was built over every document rather than the filtered subset, and `listIndexes` did not mention the option. ## The row that made it a wrong answer rather than a missing feature An over-inclusive index still answers reads correctly -- it holds a superset, never a subset -- so this was not the array-destroying class of bug. That is worth saying plainly. `unique` is where it stopped being harmless: ```js createIndex({a: 1}, {unique: true, partialFilterExpression: {t: true}}) insertMany([{a: 1, t: false}, {a: 1, t: false}]) // mongod: accepted -- neither document is in the index, so neither collides // ours: E11000 duplicate key error, dup key: {a: 1} ``` **A legal insert refused.** Unique-within-a-subset -- unique email among active accounts, unique external id among synced rows -- is the whole point of the option, so every use of it hit this. Nothing in the repository covered that row. The pinned suite is crud and aggregate; neither `e2e5.js` nor `e2e6.js` writes a partial or hashed spec. ## What is in this PR 1. **`docs/M3_INDEX_TYPES_DESIGN_REVIEW.md`** -- both features' rules measured against mongod 8.3.7: which predicates a partial filter may hold, why `sparse` and `partialFilterExpression` may not be combined (67), the four hashed refusals and the one that fires at *insert* time rather than at creation (16766), and the planner rule that decides the whole design -- > a partial index may only answer a query whose predicates *imply* its > filter, so until that test exists the safe rule is to maintain the index > and never read from it. Too few documents is the one failure worse than > no index at all. Same for hashed, mirrored: equality predicates only, everything else scans. The hash value itself is a private encoding choice -- nothing a client can observe depends on it -- which is exactly why sorts may not use the tree. 2. **The refusal.** `partialFilterExpression` is now CannotCreateIndex (67) at creation, on the same judgement `cmd_update` already states about an update spec's `sort`: "ignoring the field would be the worst of the three possible answers." The two alternatives were to keep enforcing `unique` over the wrong set, or to echo the option back from `listIndexes` while not honouring it, which is a larger lie than saying no. Two rows added to the existing index-spec refusal test, with the mutation that reddens them written beside them. ## Verification 249/249 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz, operators 125/0, positional 51/0, aggregation 70/0, pinned crud 228/63/196 unmoved, the full e2e matrix including `e2e5`/`e2e6` and `crash-fuzz` green. ## Next, in the review's order Record `tests/spec/indexes/`; then partial indexes, maintained and `unique`-enforcing with the planner declining to read from them; then hashed, equality-only; then the implication test that lets a partial index serve a read. Both need one catalog field, and `write_index_catalog`'s flags byte has spare bits, so `catalog_version` stays 1 -- the same argument the free list used for its own format change.

dev added 2 commits 2026-08-10 19:18:55 +00:00

docs: M3 design review -- partial and hashed indexes ce4ff1fd63

M3's last row names them together. They are not the same kind of gap, and the
measurement is what says so.

**Hashed is honestly missing**: `createIndex({a: "hashed"})` is refused. Wrong
code -- 2 where mongod says 67, and it has four more specific ones besides --
but the right answer, and queries keep working because there is simply no
index.

**Partial is accepted and ignored.** `createIndex` reports success,
`listIndexes` does not mention `partialFilterExpression`, and the index is
built over every document rather than the ones the filter selects.

An over-inclusive index still answers reads correctly, which is worth saying
plainly: it holds a superset, never a subset. `unique` is where that stops
being true --

  createIndex({a: 1}, {unique: true, partialFilterExpression: {t: true}})
  insertMany([{a: 1, t: false}, {a: 1, t: false}])
    mongod: accepted, neither document is in the index
    ours:   E11000 duplicate key error

-- a legal insert refused. Unique-within-a-subset is the whole point of the
option, and every use of it is an insert this server rejects.

Nothing in the repository covers that row: the pinned suite is crud and
aggregate, and neither `e2e5.js` nor `e2e6.js` writes a partial or hashed
spec. So this needs its own recorded corpus, like the positional operators and
the update operators before it.

The review measures both feature's rules -- which predicates a partial filter
may hold, why `sparse` and `partialFilterExpression` may not be combined, the
four hashed refusals and the one that fires at insert time rather than at
creation -- and argues the planner rule that matters: a partial index may only
answer a query whose predicates *imply* its filter, so until that test exists
the safe rule is to maintain the index and never read from it. Too few
documents is the one failure worse than no index at all.

Recommended order: refuse `partialFilterExpression`, record the corpus, then
implement partial, then hashed, then the implication test. Both need one
catalog field each, and `write_index_catalog`'s flags byte has spare bits, so
`catalog_version` stays 1 -- the same argument the free list used.

commands: refuse partialFilterExpression instead of ignoring it 21002528be

`createIndex({a: 1}, {partialFilterExpression: ...})` answered success, built
the index over every document, and left the option out of `listIndexes`.

An over-inclusive index still answers reads correctly -- it holds a superset,
never a subset -- so this was not the array-destroying class of bug. `unique`
is where it stopped being harmless. Measured on mongod 8.3.7:

  createIndex({a: 1}, {unique: true, partialFilterExpression: {t: true}})
  insertMany([{a: 1, t: false}, {a: 1, t: false}])
    mongod: accepted -- neither document is in the index, so neither collides
    ours:   E11000 duplicate key error, dup key: {a: 1}

A legal insert refused. Unique-within-a-subset -- unique email among active
accounts, unique external id among synced rows -- is the whole point of the
option, so every use of it hit this.

Refused at creation, which is the same judgement `cmd_update` already makes
about an update spec's `sort`: "ignoring the field would be the worst of the
three possible answers." The two alternatives here were to keep enforcing
`unique` over the wrong set, or to echo the option back from `listIndexes`
while not honouring it, which is a larger lie than saying no.

CannotCreateIndex (67), joining the five existing rows of the same test with
the mutation that reddens them written beside it. The implementation is the
rest of M3's last row and comes next, against a corpus that does not exist
yet -- nothing in this repository covered the row above, in any suite.

249/249 unit tests, pinned crud corpus unmoved at 228/63/196.

dev merged commit 21002528be into main

2026-08-10 19:19:07 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dev/MultiforaDB#10