`tests/spec/indexes/` is 42/42, so all four recorded corpora are green:
positional 51, operators 125, indexes 42, aggregate 70.
Records what implementing the row taught, including the three things the
design review got wrong. Two were already noted when the corpus was
recorded ($in is allowed; a differing filter is 86); the third is new and
cheaper than the review's version: hashed needs no flags bit, because it
belongs to a key *component* and the per-component direction byte was
already there holding 0 or 1.
Also fixes a corpus case that did not measure what it said. "equality
across numeric types" sent an int32, because a source is plain JSON and
`5.0` is `5` after JSON.parse -- it was a second copy of the case above
it. Reading sources as EJSON was tried and reverted, and the recorder now
says why: EJSON's wrapper namespace collides with the query operators
these sources are made of, so `{"$regex": "x"}` became a BSONRegExp,
`structuredClone` flattened it to `{pattern, options}`, and "a filter
using $regex is refused" silently became a filter mongod accepts. The
cross-type property is a unit test instead, which is where it belongs --
it is about how this server hashes, and mongod's hash is a different
function, so a corpus could only ever check the answer.
The case is renamed to what it does measure rather than deleted: two
documents sharing a value is still the read a hashed index exists for.
Verified: 256/256 unit tests in ReleaseFast and ReleaseSafe, 87/87 fuzz,
all four corpora 0 fail, pinned scorecard unchanged at 228/63/196, the
full e2e matrix and crash-fuzz green.
`find({a: null})` has to match a document with no `a` at all, as well as
one holding an explicit null. This server matched only the explicit one --
with or without an index -- so `{a: null}` returned one row where mongod
returns two, and `{"a.b": null}` returned none where mongod returns three.
Found by tests/spec/indexes/hashed.json, which is the last of the four
recorded corpora and the first test in this repository to ask the
question. The pinned crud+aggregate suite does not: the scorecard is
unchanged at 228/63/196 across this commit.
Two layers already believed this and only the matcher did not.
`index.build_entries` stores a missing field as null under a non-sparse
index, and `evaluate_index`'s sparse guard exists specifically to stop
this query reading an index that skipped those documents -- a guard that
was defending a behaviour that did not exist. So the fix makes three
layers agree rather than introducing a rule.
The substitution is deliberately narrow, and applies to `$eq`/`$in` and
their negations `$ne`/`$nin` only. A missing field is *not* null to
anything else: `{a: {$lt: 5}}` does not match it even though null sorts
below 5, `{a: {$exists: false}}` still has to see that there is nothing
there, and `{a: {$type: "null"}}` stays false. Collecting a null candidate
instead of substituting one would have flipped all three.
Measured against mongod 8.3.7 over a document set covering missing,
explicit null, a value, an empty array and a subdocument, with and without
an index on the path. The equality family now agrees in all four
combinations. Five neighbouring answers still differ and are none of them
touched by this commit -- four trace to one root cause, comparison
operators not being type-bracketed, and one to `{a: []}` traversed by a
dotted path. Both are recorded in PLAN §6.
M3's last row, step 4 of the index review's order. `{a: "hashed"}` was
refused with "invalid index spec" -- the right answer with the wrong code,
and the half of the row the review called honestly missing.
A hashed component is stored as a tag byte plus a 64-bit hash of the
value's *ordinary* encoded bytes. Hashing the encoding rather than the
value is what makes `{a: 5}` and `{a: 5.0}` land on the same entry for
free: `bson.encode_key` already normalizes every numeric type through
f128, because two values that compare `.eq` have to encode identically for
the tree to be a memcmp. One `encode_component` does it for entry
generation and both lookup paths, so the two sides cannot disagree -- a
component hashed on the way in and not on the way out would simply never
find anything.
Collisions are harmless, because this file's governing invariant is that
an index only generates candidates and the full filter is re-applied to
every one. The single place that would not survive one is uniqueness,
which is why `unique` is refused (16764) rather than approximated.
The planner is the mirror of the partial rule and just as conservative:
equality only. A range or a sort over a hashed component would read a band
of leaves ordered by hash, which is an arbitrary set of values, so both
are declined and the query scans. That is what leaves `find({a: {$gte:
5}})` and `find({}, {sort: {a: 1}})` correct.
The catalog needs no new field. `write_index_catalog` has always written
one byte per component and that byte has only ever held 0 or 1, so a third
value costs no format change and `catalog_version` stays 1. That is a
departure from the review, which guessed at a sixth flags bit: hashed
belongs to a *component*, and a compound index may hold one beside range
ones.
Measured on mongod 8.3.7 rather than recalled, and three of the five
answers were not what the corpus source assumed:
two hashed components 31303, codeName Location31303
unique on a hashed index 16764, codeName Location16764
an unknown plugin string 67, codeName CannotCreateIndex
an array at the path 16766 -- a *writeError* beside `ok: 1` on an
insert or update, and a command error from
createIndexes over data that already holds one
an array through a path refused for a *one-element* array too, which
is why `array_on_path` walks the path instead
of counting the values at it
tests/spec/indexes/hashed.json goes 0/18 -> 17/18. The one that remains
is not about hashed indexes: `find({a: null})` has to match a document
with no `a`, and this server matches only an explicit null -- with or
without an index. Next commit.
