plan: M2 splits in two (amendment A6)

The design review found the milestone's gate names a corpus that is not there:
`mongodb/specifications` has no aggregation suite, and the thirteen
`aggregate-*.json` files this project runs live inside `crud` and test the
aggregate *command*, not stages. One name was covering two milestones.

M2 becomes the command surface -- $out, $merge, db.aggregate(), collation,
let -- gated on the corpus that already exists, at 0 fail. M2.5 becomes the
engine: expression evaluator, per-stage document iterator, accumulators,
$unwind; $lookup and $facet explicitly out of the first cut, gated on a
corpus this project writes with every expectation measured against mongod.

Command surface first, engine second, chosen with the cost stated: the engine
is what gives wrong answers now -- six of eight probed pipelines answer ok:1
with a wrong result -- so this order leaves them alive a milestone longer.
Which is why M2 carries the refusals: every unimplemented construct stops
answering 0 and starts answering an error with a measured code, the same move
expectEvents made before the free list in M1. It will push the scorecard down,
and that is the point.

The review keeps its pre-decision recommendation verbatim, so the argument the
decision overrode stays legible.
This commit is contained in:
A.Shakhmatov
2026-08-09 19:40:03 +03:00
parent 37211a6cda
commit 7ae81b8179
2 changed files with 75 additions and 4 deletions

61
PLAN.md
View File

@@ -394,6 +394,55 @@ per-collection gate judges what reclamation left. Same six rounds, 256 MB.
Numbers and reproduction in `tests/e2e/results/m0-gates.txt` under `[M1.1]` and
`[M1.2]`.
### Amendment A6 — M2 splits in two, and its gate did not exist (amends the M2 row)
The M2 design review (`docs/M2_DESIGN_REVIEW.md`) found that the milestone's
gate names a corpus that is not there. `mongodb/specifications` has no
aggregation suite: the thirteen `aggregate-*.json` files this project runs live
inside `crud` and test the aggregate *command* — cursor and `batchSize`,
`readConcern` routing, the write semantics of `$out`/`$merge`, `collation`,
`let`. They touch stages barely and expressions not at all; `$lookup`,
`$unwind`, `$facet`, `$addFields` and `$replaceRoot` appear nowhere in the
pinned corpus. A literal "green" is unreachable besides, since 10 of the skips
are version- or topology-gated and cannot pass on a standalone.
So one milestone name was covering two milestones, and they are split.
**M2 is the command surface**, gated on the corpus that already exists: `$out`,
`$merge`, `db.aggregate()`, collation, `let`. That is 8 of the 13 current
failures in the write stages alone. It is small and fully measurable today.
**M2.5 is the engine**, gated on a corpus this project has to write, in the
unified format the runner already reads, with every expectation measured
against mongod 8.3.7 rather than recalled — the discipline that corrected three
assumptions in M1's session work. The tiers, in dependency order: the
expression evaluator, then a per-stage document iterator (the stream is a
materialized window today, which cannot express 1→N), then the accumulators.
`$unwind` is in the first cut because it falls out of the iterator for free.
`$lookup` is out: a stage that takes a second collection's lock while the
pipeline holds one is a lock-order question that should not ride along.
`$facet` is out: sub-pipeline execution should be allowed to settle after the
iterator exists.
**Sequencing: command surface first, engine second** — chosen deliberately, and
with the cost stated rather than glossed. The engine is what gives wrong
answers *now*: measured against a live server, `$avg`, `$max` and `$push` all
return `0`, a compound `_id` collapses every document into one group keyed by
the unevaluated expression, `$literal` is echoed back, and a `$match` after a
`$project` still sees the projected-away field. Six of eight probed pipelines
answer `ok: 1` with a wrong result; only `$addFields` fails honestly. Doing the
command surface first leaves those alive for a further milestone.
**Which is why M2 carries the refusals.** Every construct the engine does not
implement stops answering `0` and starts answering an error, with the code
measured against mongod: unknown accumulators, non-path expressions where a
path is expected, `$project` anywhere but last. This is the same move
`expectEvents` made before the free list in M1 — it will push the scorecard
*down*, and that is the point. An unrecognised stage is a bug report; an `$avg`
that returns `0` is a corrupted report nobody files. Without this the sequencing
above would be trading a measurable milestone for a year of silent wrong
answers, and the trade is only acceptable because the lie is removed first.
---
## 3. Milestones and gates
@@ -402,7 +451,8 @@ Numbers and reproduction in `tests/e2e/results/m0-gates.txt` under `[M1.1]` and
|---|---|---|---|
| M0 | **mmap + WAL foundation** | data file format, page/extent allocator, mmap slab + B+tree arena, copy-on-write + page free list (A1/A2), watermark/replay, checkpoint (= compaction repurposed), leaf payload → slab offset (A3), drop docs hashmap, churn measurement | D7 (6 items) |
| M1 | **Cursors + wire polish** | getMore / killCursors / batchSize; server-side cursor state with idle timeout; sessions plumbing (lsid accepted) as drivers send it; hello advertisement updates; **`moreToCome` on requests** (see the bug below); command-monitoring assertions in the spec runner | crud spec suite green; e2e green |
| M2 | **Aggregation expansion** | pipeline stages and expression engine (tiered scope defined at M2 design review) | aggregate spec suite green |
| M2 | **The `aggregate` command surface** | `$out`, `$merge`, `db.aggregate()` (`{aggregate: 1}`), collation, `let`; plus refusing every pipeline construct the engine does not implement instead of answering `0` (amendment A6) | `aggregate-*.json` in the crud corpus: 0 fail |
| M2.5 | **The aggregation engine** | expression evaluator, per-stage document iterator, the accumulators, `$unwind`; `$lookup`/`$facet` explicitly out of the first cut (amendment A6) | a purpose-built stage corpus, every expectation measured against mongod |
| M3 | **Update operators + index types** | $setOnInsert, $addToSet, $mul, $min/$max, $pop, $pullAll, $currentDate, pipeline updates; partial + hashed indexes | remaining crud coverage; e2e3/e2e4 green |
| M4 | **Sessions + transactions** | logical sessions, snapshot isolation on the mmap engine, write concern at commit | sessions + transactions spec suites green |
| M5 | **Change streams** | change feed + resume tokens (likely log-seq based), getMore integration | change-streams spec suite green |
@@ -969,8 +1019,13 @@ has to be its own commit with its own re-recorded scorecard.
taking a `session` argument, and no `lsid` assertion. The value here is
protocol hygiene, not a number — and after Stage 1 a wrongly-refused command
would have shown up as a changed event stream rather than silently.
- **M2 aggregation**: stage/expression tiers, which spec-test files are
the gate, whether $lookup/$unwind/facet make the first cut.
- **M2 aggregation** — *settled, see amendment A6 and `docs/M2_DESIGN_REVIEW.md`.*
The three questions this entry held are answered: the tiers, the gate (the
named one does not exist), and the first cut ($unwind in, $lookup and $facet
out). What the review did *not* settle and the engine milestone must:
`$out`/`$merge` durability semantics, whether the expression evaluator is
shared with M3's pipeline updates, and whether `allowDiskUse` has to stop
being a lie.
- **M4 transactions**: snapshot isolation over mmap (COW vs undo), read
concern snapshot, conflict → TransientTransactionError semantics,
retryable-writes interplay.