`tests/spec/indexes/` is 42/42, so all four recorded corpora are green:
positional 51, operators 125, indexes 42, aggregate 70.
Records what implementing the row taught, including the three things the
design review got wrong. Two were already noted when the corpus was
recorded ($in is allowed; a differing filter is 86); the third is new and
cheaper than the review's version: hashed needs no flags bit, because it
belongs to a key *component* and the per-component direction byte was
already there holding 0 or 1.
Also fixes a corpus case that did not measure what it said. "equality
across numeric types" sent an int32, because a source is plain JSON and
`5.0` is `5` after JSON.parse -- it was a second copy of the case above
it. Reading sources as EJSON was tried and reverted, and the recorder now
says why: EJSON's wrapper namespace collides with the query operators
these sources are made of, so `{"$regex": "x"}` became a BSONRegExp,
`structuredClone` flattened it to `{pattern, options}`, and "a filter
using $regex is refused" silently became a filter mongod accepts. The
cross-type property is a unit test instead, which is where it belongs --
it is about how this server hashes, and mongod's hash is a different
function, so a corpus could only ever check the answer.
The case is renamed to what it does measure rather than deleted: two
documents sharing a value is still the read a hashed index exists for.
Verified: 256/256 unit tests in ReleaseFast and ReleaseSafe, 87/87 fuzz,
all four corpora 0 fail, pinned scorecard unchanged at 228/63/196, the
full e2e matrix and crash-fuzz green.
M3's last row names them together. They are not the same kind of gap, and the
measurement is what says so.
**Hashed is honestly missing**: `createIndex({a: "hashed"})` is refused. Wrong
code -- 2 where mongod says 67, and it has four more specific ones besides --
but the right answer, and queries keep working because there is simply no
index.
**Partial is accepted and ignored.** `createIndex` reports success,
`listIndexes` does not mention `partialFilterExpression`, and the index is
built over every document rather than the ones the filter selects.
An over-inclusive index still answers reads correctly, which is worth saying
plainly: it holds a superset, never a subset. `unique` is where that stops
being true --
createIndex({a: 1}, {unique: true, partialFilterExpression: {t: true}})
insertMany([{a: 1, t: false}, {a: 1, t: false}])
mongod: accepted, neither document is in the index
ours: E11000 duplicate key error
-- a legal insert refused. Unique-within-a-subset is the whole point of the
option, and every use of it is an insert this server rejects.
Nothing in the repository covers that row: the pinned suite is crud and
aggregate, and neither `e2e5.js` nor `e2e6.js` writes a partial or hashed
spec. So this needs its own recorded corpus, like the positional operators and
the update operators before it.
The review measures both feature's rules -- which predicates a partial filter
may hold, why `sparse` and `partialFilterExpression` may not be combined, the
four hashed refusals and the one that fires at insert time rather than at
creation -- and argues the planner rule that matters: a partial index may only
answer a query whose predicates *imply* its filter, so until that test exists
the safe rule is to maintain the index and never read from it. Too few
documents is the one failure worse than no index at all.
Recommended order: refuse `partialFilterExpression`, record the corpus, then
implement partial, then hashed, then the implication test. Both need one
catalog field each, and `write_index_catalog`'s flags byte has spare bits, so
`catalog_version` stays 1 -- the same argument the free list used.
Scorecard 204 -> 218 pass, 87 -> 73 fail: the fourteen `arrayFilters` cases
across five files, which is the whole of what the two implementation commits
were expected to move and nothing else. Positional corpus 51/51.
The three divergences measured on the way are written into PLAN §6 rather
than left in commit messages: `$` with two predicates on one array that no
element satisfies together, an array filter with a top-level `$and`/`$or`,
and a literal index into a scalar element -- the last being the one place the
positional walk and the plain indexed path now answer differently, which is
worth a commit of its own and needs its own measurements first.
The design review gets an outcome note, since two of its guesses were wrong
and the corpus is where that was settled.
The plan names a missing feature sized at 14 corpus cases. The measurement
found a data-loss bug.
`{$set: {'y.$[i].b': 2}}` does not fail to update the array -- it replaces
the array with `{"$[i]": {"b": 2}}` and answers ok: 1, modifiedCount: 1.
Every element is discarded. `src/update.zig:283` reaches a non-numeric
segment under an array and takes the "treat as non-array: replace with a
doc" branch, so `$`, `$[]` and `$[<ident>]` all destroy what they were meant
to descend into, under every operator -- `$inc` through `$[i]` stores the
operand rather than incrementing. 16 of 17 probe cases diverge from mongod.
`arrayFilters` is not implicated: the string appears nowhere in `src/`. The
option is accepted off the wire and dropped.
The gate finding, which is the reason to write this before any code: M3's
gate is "remaining crud coverage; e2e3/e2e4 green". e2e3 and e2e4 contain
zero positional paths and would stay green through every version of this
bug. The pinned corpus covers `$[<ident>]` only -- it has no `$[]` case and
no bare `$` case anywhere. A fix scoped to what the gate measures would go
green on 14 new passes with two of the three ways to destroy an array still
live. The gate would certify the bug as fixed.
Also measured, none of it guessable and two of them inverting how set_path
behaves today: one path can name many targets (`y.$[].c.$[].d` is a genuine
cross-product); a positional segment never creates anything, so a missing
or non-array path is an error where `$set: {'a.b': 1}` would construct, and
upsert gets no special case; ten distinct refusals across codes 2, 9 and 14,
recorded with their messages, including one row that breaks the otherwise
tidy 2/9 split and is left untidy on purpose.
Recommends refusing first as its own commit -- the M2 doctrine, and it stops
the data loss without waiting on the redesign -- then a recorded positional
corpus built like tests/spec/aggregate/, with `$[]` and `$[<ident>]`
together and `$` after, since only `$` needs the matched index carried out
of query evaluation.
The seven reachable failures of M2, and the first commit of the milestone to
move the scorecard: 194/97/196 -> 201/90/196, with `aggregate-*.json` going
9 pass / 13 fail to 16 pass / 6 fail. The seven that moved are exactly the
seven priced as reachable, and the six that remain are exactly the six
attributed to M2.5 ($addFields, the expression engine), M4 ($listLocalSessions)
and M8 (collation).
Both stages write to a collection the pipeline is not reading, and three things
stood against doing that in the handler: `aggregate` is a `.read` command,
dispatch takes locks from a static table keyed on the command name before the
handler runs, and `Collection.lock` allows exactly one collection lock at a
time. So the handler computes the output under the locks it has and leaves it
in `Context.pending_write`; the epilogue applies it with nothing held, beside
the commit and the checkpoint already there. The `.read`/`.write` contract is
amended in its own comment rather than quietly broken.
`pending_write` is cleared at the top of every dispatch, so a handler that
errors before setting one cannot leave the previous command's write to fire. A
failed write replaces the pipeline's `ok: 1` with the failure, because a client
told the aggregation succeeded would believe the collection had been written.
What the stages do not implement is refused, not ignored: `$merge`'s
`whenMatched`, `whenNotMatched`, `on` and `let` all select behaviour this
server does not have, and a `whenMatched: "fail"` that silently merged would be
the same lie Tier 0 spent three commits removing. Codes measured against
mongod 8.3.7. `$out` and `$merge` answer byte-identically to it on both the
replace and the upsert case.
NOT ATOMIC, and said out loud in the code rather than left to be discovered.
mongod replaces an `$out` target atomically; this engine has no
cross-collection atomicity and no rename to build one from, so a crash between
the drop and the last insert leaves the target holding part of the new output
where MongoDB would leave the whole of the old. The fix is
write-to-temp-and-rename and rename is a command that does not exist here.
The test's mutation is the argument for the epilogue in one line: apply the
write inside the `$out` branch and it deadlocks rather than fails.
191/191 unit tests in ReleaseFast and ReleaseSafe, 83/83 fuzz, e2e 49, e2e2
concurrent, e2e3 16, e2e4 17, e2e6 72, e2e7 86.
Priced the 13 aggregate failures one by one instead of counting them. Six
cannot turn green in M2 whatever is built: three need $listLocalSessions (M4)
*and* $addFields (M2.5), two need the expression engine, one needs a collation
(M8). So the gate reads '0 fail among the seven reachable', with the other six
named and attributed.
Note what the three db.aggregate() cases actually need. Implementing
{aggregate: 1} moves none of them, because each then fails on
$listLocalSessions instead. The review priced that line 'orthogonal, and
cheap'; it was cheap and worth nothing.
The seven that remain are $out and $merge, and they need a decision the review
had no authority to take. They write to a collection the pipeline is not
reading, and three things stand against that: aggregate is declared .read and
the dispatch contract says only .write commands may mutate; dispatch acquires
locks from a static table keyed on the command name, before the handler runs;
and Collection.lock's own comment states the invariant that decides it -- never
more than one collection lock at a time.
That is the same objection that kept $lookup out of M2.5's first cut, and the
review failed to apply it to the write stages in the same breath. Recorded as
the review's own error rather than quietly corrected. Two options set out in
§7, plus the durability question neither of them answers: mongod's $out
replaces the target atomically and this engine has no cross-collection
atomicity.
Nothing past Tier 0 is implemented until one is chosen.
The scorecard collapses a whole-file skip into a single `*` line, so counting
the reason lines undercounted the cases they cover. Every one of the 19 is
version- or topology-gated, which is why the gate reads *0 fail* rather than
*all pass*.
The design review found the milestone's gate names a corpus that is not there:
`mongodb/specifications` has no aggregation suite, and the thirteen
`aggregate-*.json` files this project runs live inside `crud` and test the
aggregate *command*, not stages. One name was covering two milestones.
M2 becomes the command surface -- $out, $merge, db.aggregate(), collation,
let -- gated on the corpus that already exists, at 0 fail. M2.5 becomes the
engine: expression evaluator, per-stage document iterator, accumulators,
$unwind; $lookup and $facet explicitly out of the first cut, gated on a
corpus this project writes with every expectation measured against mongod.
Command surface first, engine second, chosen with the cost stated: the engine
is what gives wrong answers now -- six of eight probed pipelines answer ok:1
with a wrong result -- so this order leaves them alive a milestone longer.
Which is why M2 carries the refusals: every unimplemented construct stops
answering 0 and starts answering an error with a measured code, the same move
expectEvents made before the free list in M1. It will push the scorecard down,
and that is the point.
The review keeps its pre-decision recommendation verbatim, so the argument the
decision overrode stays legible.
PLAN §3 gives M2 one line of scope and one line of gate, and §6 lists the three
questions it deferred. This answers them, and reports one finding that has to
be settled before the rest is worth discussing: the gate named in the plan does
not exist. `mongodb/specifications` has no aggregation suite -- the thirteen
`aggregate-*.json` files we run live inside `crud` and test the aggregate
*command* surface, not stages. $lookup, $unwind, $facet, $addFields and
$replaceRoot appear nowhere in the pinned corpus.
Measured, not recalled. Seven stages exist, not the nine an earlier note in
this project claimed -- $set and $unset are update operators. There is no
expression engine: $field paths are open-coded in two places inside run_group
and $sum is the only accumulator. Probed on a live server, six of eight
pipelines answer ok:1 with a wrong result -- $avg, $max and $push all return
0, a compound _id groups everything into one bucket keyed by the unevaluated
expression, $literal is echoed back, and a $match after a $project still sees
the projected-away field. Only $addFields fails honestly.
That drives the tiering. M2 is not "add stages to the chain": the stream is a
materialized window and each stage moves its bounds, which cannot express
1->N ($unwind), a second collection ($lookup), or sub-pipelines ($facet), and
$project is not a stage transform at all. Six tiers proposed, four gate options
priced, and a recommendation: Tier 0 (refuse what is not implemented) alone
first, for the same reason expectEvents preceded the free list in M1 -- it will
move the scorecard down, and that is the point.
No decision is recorded in PLAN.md yet; that is what the review is for.
Wrap signatures and long expressions to the 100-column limit and make every
file zig fmt clean. Semantics-preserving throughout: ignoring whitespace and
the trailing commas that wrapping introduces, every file here is byte-identical
to its predecessor, and the one apparent exception is a warning string split
with `++`, which concatenates at comptime to the same bytes.
src/index.zig and src/commands.zig are reformatted in the commits that follow,
because their reformat is interleaved with in-flight changes to them and
separating the two would need the reformat re-derived rather than moved.
