dev/MultiforaDB
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: M3 design review -- partial and hashed indexes

Browse Source 
M3's last row names them together. They are not the same kind of gap, and the
measurement is what says so.

**Hashed is honestly missing**: `createIndex({a: "hashed"})` is refused. Wrong
code -- 2 where mongod says 67, and it has four more specific ones besides --
but the right answer, and queries keep working because there is simply no
index.

**Partial is accepted and ignored.** `createIndex` reports success,
`listIndexes` does not mention `partialFilterExpression`, and the index is
built over every document rather than the ones the filter selects.

An over-inclusive index still answers reads correctly, which is worth saying
plainly: it holds a superset, never a subset. `unique` is where that stops
being true --

  createIndex({a: 1}, {unique: true, partialFilterExpression: {t: true}})
  insertMany([{a: 1, t: false}, {a: 1, t: false}])
    mongod: accepted, neither document is in the index
    ours:   E11000 duplicate key error

-- a legal insert refused. Unique-within-a-subset is the whole point of the
option, and every use of it is an insert this server rejects.

Nothing in the repository covers that row: the pinned suite is crud and
aggregate, and neither `e2e5.js` nor `e2e6.js` writes a partial or hashed
spec. So this needs its own recorded corpus, like the positional operators and
the update operators before it.

The review measures both feature's rules -- which predicates a partial filter
may hold, why `sparse` and `partialFilterExpression` may not be combined, the
four hashed refusals and the one that fires at insert time rather than at
creation -- and argues the planner rule that matters: a partial index may only
answer a query whose predicates *imply* its filter, so until that test exists
the safe rule is to maintain the index and never read from it. Too few
documents is the one failure worse than no index at all.

Recommended order: refuse `partialFilterExpression`, record the corpus, then
implement partial, then hashed, then the implication test. Both need one
catalog field each, and `write_index_catalog`'s flags byte has spare bits, so
`catalog_version` stays 1 -- the same argument the free list used.
This commit is contained in:
A.Shakhmatov
2026-08-10 22:09:17 +03:00
parent 98fde82946
commit ce4ff1fd63
1 changed files with 159 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

159
docs/M3_INDEX_TYPES_DESIGN_REVIEW.md Normal file
View File

@@ -0,0 +1,159 @@
# M3 design review — partial and hashed indexes
The last line of M3's row. Written before any code, like the M2 and
`arrayFilters` reviews, and for the same reason: both times the measurement
disagreed with the plan before the implementation did.
Everything below was measured on 2026-08-10 against mongod 8.3.7 on `:27099`
and this server at `98fde82` on `:27020`, running the identical probe against
both.
---
## 1. The two halves are not the same kind of gap
PLAN §3 names them together — "partial + hashed indexes" — as though they were
one item. They are not.
**Hashed is honestly missing.** `createIndex({a: "hashed"})` is refused:
```
mongod: a_hashed
ours: ERR [2] invalid index spec
```
Wrong code (mongod says 67 for an unknown key direction, and has four more
specific ones besides), but the right answer. A client is told no and knows
where it stands. Queries against the collection keep working, because there is
simply no index.
**Partial is accepted and ignored.**
```
mongod: listIndexes -> {"key":{"a":1},"name":"a_1","partialFilterExpression":{"a":{"$gte":5}}}
ours: listIndexes -> {"key":{"a":1},"name":"a_1"}
```
`createIndex` answers `a_1` and reports success. The index is built over
*every* document rather than the ones the filter selects, `listIndexes` does
not mention the option, and nothing tells the client that the index they asked
for is not the index they have.
## 2. Ignoring it is not harmless, and here is the row that proves it
An over-inclusive index still answers queries correctly — it contains a
superset of what it should, never a subset, so a scan through it finds
everything. Reads are safe. That is worth saying plainly, because it is the
reason this is a smaller fire than `arrayFilters` was.
It is not, however, harmless, and `unique` is where it stops being harmless:
```js
// createIndex({a: 1}, {unique: true, partialFilterExpression: {t: true}})
// insertMany([{a: 1, t: false}, {a: 1, t: false}])
mongod: accepted -- neither document is in the index, so neither collides
ours: ERR [11000] E11000 duplicate key error ... dup key: {a: 1}
```
**A legal insert is refused.** The whole point of a partial unique index is
uniqueness *within a subset*: unique email among active accounts, unique
external id among synced rows. Every one of those is an insert this server
rejects and mongod accepts. The client sees a duplicate-key error naming a
constraint it deliberately scoped away.
Two smaller divergences ride along: `listIndexes` reports an index
specification that is not the one on disk, so a client comparing specs before
creating decides wrongly; and the index carries every document, which is the
storage the option existed to avoid.
## 3. What the gate can see
Nothing. There is no index-management corpus here — the pinned suite this
repo fetches is crud + aggregate, and `tests/e2e/e2e5.js` and `e2e6.js` test
indexes but neither writes a partial or hashed spec. The `unique`-plus-partial
row above is not covered by a single test in the repository, in any suite.
So M3's last line needs its own recorded corpus, exactly like the positional
operators and the update operators did. `tests/spec/indexes/`, same shape:
inputs authored in `sources/`, expectations measured from mongod, run through
`run.js --suite-dir`.
## 4. What the fix has to be, measured
### Partial
The filter is not an arbitrary query. mongod restricts it, and the restriction
is what makes the planner's job possible:
| | mongod |
|---|---|
| equality, `$gt`/`$gte`/`$lt`/`$lte`, `$exists: true`, `$type` | allowed |
| `$and`, and `$or` from 8.0 | allowed |
| `$regex` | `Error in specification ... ` (67) |
| `partialFilterExpression` that is not a document | TypeMismatch (14) |
| `partialFilterExpression` **with `sparse`** | (67) — the two may not be combined |
The planner rule is the load-bearing one and it is not symmetric: a partial
index may only answer a query whose predicates *imply* the filter. Using it
for a query it does not cover returns too few documents, which is the one
failure mode worse than not using an index at all. Until that implication test
exists, the safe planner rule is **never use a partial index for a read** —
maintain it, enforce `unique` through it, and let reads fall back to a scan.
That is a correct server that has not yet earned the speedup, and it is where
this should land first.
### Hashed
| | mongod |
|---|---|
| `{a: "hashed"}` | allowed, name `a_hashed` |
| `{a: "hashed", b: 1}` | allowed — one hashed component beside range ones |
| `{a: "hashed", b: "hashed"}` | 31303, "A maximum of one index field is allowed to be hashed" |
| `unique` on a hashed index | 16764, "Currently hashed indexes cannot guarantee uniqueness" |
| an array value at the hashed path | 16766, **at insert time**, not at creation |
| `{a: "bogus"}` | 67, "Unknown index plugin" |
A hashed index answers equality only; mongod still returns correct results for
a range query or a sort over a hashed field by not using the index. So the
planner rule is the mirror of the partial one and just as conservative:
equality predicates only, everything else scans.
The hash itself need not match mongod's. Nothing a client can observe depends
on the value — `listIndexes` reports `"hashed"`, not a hash — so this is a
private encoding choice, and the ordering of the tree is then meaningless,
which is exactly why sorts may not use it.
## 5. The order this should land in
The same shape the `arrayFilters` work took, and for the same reason: the
refusal is small, correct on its own, and stops the wrong answer before the
corpus that measures it exists.
1. **Refuse `partialFilterExpression`.** This repo already has the precedent
and states it in `cmd_update`: an update spec's `sort` is refused because
"ignoring the field would be the worst of the three possible answers — the
client asked for a specific document and would silently get a different
one." Identical logic. The two alternatives are to keep enforcing `unique`
over the wrong set, or to report the option in `listIndexes` while not
honouring it, which is a larger lie than the current one.
2. **Record `tests/spec/indexes/`** against mongod: partial creation rules,
the `unique` subset semantics, hashed creation rules, and the read paths
for both. Red by construction.
3. **Partial indexes**, maintained and `unique`-enforcing, planner declining
to read from them.
4. **Hashed indexes**, equality-only in the planner.
5. **The implication test**, which is what lets a partial index serve a read,
and is a planner change rather than an index one.
Steps 3 and 4 both need a catalog field. `write_index_catalog` has a `flags`
byte with four bits used, so a fifth can mean "a partial filter follows" and a
sixth "this key is hashed" — old files never set them and read back
identically, so `catalog_version` stays 1. That is the same argument the free
list used for its own format change and it holds here for the same reason.
## 6. Not covered
Neither `$or` in a partial filter beyond accepting it, nor `2dsphere`, `text`,
`wildcard` or `collation` — none of them is in M3's row, and each is a
milestone-sized item that would arrive with its own review.