M3: hashed indexes #13
Reference in New Issue
Block a user
Delete Branch "m3-hashed-indexes"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Step 4 of
docs/M3_INDEX_TYPES_DESIGN_REVIEW.md's order, and the lastindex type in M3's row.
tests/spec/indexes/goes 24/42 to 42/42.A hashed key holds a hash of its value
{a: "hashed"}was refused with "invalid index spec" -- the right answerwith the wrong code, and the half of the row the review called honestly
missing.
A hashed component stores a tag byte plus a 64-bit hash of the value's
ordinary encoded bytes. Hashing the encoding rather than the value is
what makes
{a: 5}and{a: 5.0}one entry for free:bson.encode_keyalready normalizes every numeric type through f128, because two values
that compare
.eqhave to encode identically for the tree to be amemcmp. One
encode_componentserves entry generation and both lookuppaths, so the two sides cannot disagree -- a component hashed on the way
in and not on the way out would simply never find anything.
Collisions are harmless: an index generates candidates and the full
filter is re-applied to every one. The single place that would not
survive one is uniqueness, which is why
uniqueis refused (16764)rather than approximated.
The planner is the mirror of the partial rule and just as conservative:
equality only. A range or a sort would read a band of leaves ordered by
hash, which is an arbitrary set of values, so both are declined and the
query scans -- which is what leaves
find({a: {$gte: 5}})andfind({}, {sort: {a: 1}})correct.The catalog needs no new field, and the review was wrong about why
The review proposed a sixth flags bit. Hashed is a property of a key
component, not of an index -- a compound index may hold one hashed
component beside range ones -- and the per-component direction byte has
always been written and has only ever held 0 or 1. A third value costs no
format change and
catalog_versionstays 1.Measured, not recalled
Three answers were not what the corpus source assumed:
codeNamefor 31303 / 16764Location31303/Location16764-- bare location numberscodeNamefor{a: "bogus"}CannotCreateIndex, the named 67ok: 1, so the rest of the batch landscreateIndexesWhat the corpus caught that was not about indexes
find({a: null})has to match a document with noa. This servermatched only an explicit null -- with or without an index -- and
{"a.b": null}matched nothing at all.Two layers already believed otherwise and only the matcher did not:
index.build_entriesstores a missing field as null under a non-sparseindex, and
evaluate_index's sparse guard exists specifically to stopthis query reading an index that skipped those documents. The guard was
defending a behaviour that did not exist.
Fixed narrowly, for
$eq/$inand their negations only. A missing fieldis not null to
$lt(even though null sorts below 5), to$exists: false, or to$type: "null"-- collecting a null candidateinstead of substituting one would have flipped all three. The pinned
crud+aggregate scorecard did not move: it does not cover the question.
Three neighbours were measured at the same time and left alone, each
recorded in PLAN §6 as its own item: comparison operators are not
type-bracketed (one root cause, four visible divergences); a dotted path
through an empty array is treated as absent; and a key-pattern direction
may be any non-zero, non-NaN number, echoed back verbatim.
A corpus case that did not measure what it said
"equality across numeric types" sent an int32, because a source is plain
JSON and
5.0is5afterJSON.parse-- it was a second copy of thecase above it. Reading sources as EJSON was tried and reverted, and the
recorder now records why: EJSON's wrapper namespace collides with the
query operators these sources are made of, so
{"$regex": "x"}became aBSONRegExp,structuredCloneflattened it to{pattern, options}, and"a filter using $regex is refused" silently became a filter mongod
accepts. The cross-type property is a unit test instead, which is where
it belongs -- it is about how this server hashes, and mongod's hash is a
different function.
Verification
tests/spec/indexes/42/0,operators/125/0,positional/51/0,aggregate/70/0crash-fuzzgreenLeft open in M3: step 5, the implication test that lets a partial index
serve a read. A planner change, not an index one.
M3's last row, step 4 of the index review's order. `{a: "hashed"}` was refused with "invalid index spec" -- the right answer with the wrong code, and the half of the row the review called honestly missing. A hashed component is stored as a tag byte plus a 64-bit hash of the value's *ordinary* encoded bytes. Hashing the encoding rather than the value is what makes `{a: 5}` and `{a: 5.0}` land on the same entry for free: `bson.encode_key` already normalizes every numeric type through f128, because two values that compare `.eq` have to encode identically for the tree to be a memcmp. One `encode_component` does it for entry generation and both lookup paths, so the two sides cannot disagree -- a component hashed on the way in and not on the way out would simply never find anything. Collisions are harmless, because this file's governing invariant is that an index only generates candidates and the full filter is re-applied to every one. The single place that would not survive one is uniqueness, which is why `unique` is refused (16764) rather than approximated. The planner is the mirror of the partial rule and just as conservative: equality only. A range or a sort over a hashed component would read a band of leaves ordered by hash, which is an arbitrary set of values, so both are declined and the query scans. That is what leaves `find({a: {$gte: 5}})` and `find({}, {sort: {a: 1}})` correct. The catalog needs no new field. `write_index_catalog` has always written one byte per component and that byte has only ever held 0 or 1, so a third value costs no format change and `catalog_version` stays 1. That is a departure from the review, which guessed at a sixth flags bit: hashed belongs to a *component*, and a compound index may hold one beside range ones. Measured on mongod 8.3.7 rather than recalled, and three of the five answers were not what the corpus source assumed: two hashed components 31303, codeName Location31303 unique on a hashed index 16764, codeName Location16764 an unknown plugin string 67, codeName CannotCreateIndex an array at the path 16766 -- a *writeError* beside `ok: 1` on an insert or update, and a command error from createIndexes over data that already holds one an array through a path refused for a *one-element* array too, which is why `array_on_path` walks the path instead of counting the values at it tests/spec/indexes/hashed.json goes 0/18 -> 17/18. The one that remains is not about hashed indexes: `find({a: null})` has to match a document with no `a`, and this server matches only an explicit null -- with or without an index. Next commit.`find({a: null})` has to match a document with no `a` at all, as well as one holding an explicit null. This server matched only the explicit one -- with or without an index -- so `{a: null}` returned one row where mongod returns two, and `{"a.b": null}` returned none where mongod returns three. Found by tests/spec/indexes/hashed.json, which is the last of the four recorded corpora and the first test in this repository to ask the question. The pinned crud+aggregate suite does not: the scorecard is unchanged at 228/63/196 across this commit. Two layers already believed this and only the matcher did not. `index.build_entries` stores a missing field as null under a non-sparse index, and `evaluate_index`'s sparse guard exists specifically to stop this query reading an index that skipped those documents -- a guard that was defending a behaviour that did not exist. So the fix makes three layers agree rather than introducing a rule. The substitution is deliberately narrow, and applies to `$eq`/`$in` and their negations `$ne`/`$nin` only. A missing field is *not* null to anything else: `{a: {$lt: 5}}` does not match it even though null sorts below 5, `{a: {$exists: false}}` still has to see that there is nothing there, and `{a: {$type: "null"}}` stays false. Collecting a null candidate instead of substituting one would have flipped all three. Measured against mongod 8.3.7 over a document set covering missing, explicit null, a value, an empty array and a subdocument, with and without an index on the path. The equality family now agrees in all four combinations. Five neighbouring answers still differ and are none of them touched by this commit -- four trace to one root cause, comparison operators not being type-bracketed, and one to `{a: []}` traversed by a dotted path. Both are recorded in PLAN §6.`tests/spec/indexes/` is 42/42, so all four recorded corpora are green: positional 51, operators 125, indexes 42, aggregate 70. Records what implementing the row taught, including the three things the design review got wrong. Two were already noted when the corpus was recorded ($in is allowed; a differing filter is 86); the third is new and cheaper than the review's version: hashed needs no flags bit, because it belongs to a key *component* and the per-component direction byte was already there holding 0 or 1. Also fixes a corpus case that did not measure what it said. "equality across numeric types" sent an int32, because a source is plain JSON and `5.0` is `5` after JSON.parse -- it was a second copy of the case above it. Reading sources as EJSON was tried and reverted, and the recorder now says why: EJSON's wrapper namespace collides with the query operators these sources are made of, so `{"$regex": "x"}` became a BSONRegExp, `structuredClone` flattened it to `{pattern, options}`, and "a filter using $regex is refused" silently became a filter mongod accepts. The cross-type property is a unit test instead, which is where it belongs -- it is about how this server hashes, and mongod's hash is a different function, so a corpus could only ever check the answer. The case is renamed to what it does measure rather than deleted: two documents sharing a value is still the read a hashed index exists for. Verified: 256/256 unit tests in ReleaseFast and ReleaseSafe, 87/87 fuzz, all four corpora 0 fail, pinned scorecard unchanged at 228/63/196, the full e2e matrix and crash-fuzz green.