commands: distinct #4

dev · 2026-08-10T14:42:21Z

dev commented

2026-08-10 14:42:21 +00:00

M3 opens with the cheapest lever: a whole command that did not exist. Five
corpus cases answered "no such command".

Two things were measured against mongod 8.3.7 rather than recalled, and the
first is not what anyone would guess:

The answer is sorted in canonical BSON order, not in the order the
values were met. {s: "b"}, {s: "a"}, {s: null} answers [null, "a", "b"].
Insertion order is the obvious implementation, it passes every test anybody
would think to write by hand against [11, 22, 33], and it is wrong.
Deduping is the same comparator, so an int32 1 and a double 1.0 collapse
while null and "1" survive.

Both fall out of bson.compare, which $sort and $min already use -- and
that is not luck: mongod accumulates into a BSONElementSet ordered by the
same woCompare. The rest reuses the shared read path.

Running the identical probe against both servers now agrees on every
semantic row.

Scorecard

201 pass / 90 fail -> 204 / 87. distinct.json 0/2 -> 2/0,
distinct-rawdata 0/1 -> 1/0. distinct-comment nets zero: its "no such
command" is replaced by the pre-4.4.14 document-comment case, which
estimatedDocumentCount already carries as a standing failure -- emulating a
bug fixed in 4.4.14 for one command would make the two disagree.
distinct-collation still needs M8, but now fails with the honest
"expected 1 elements, got 2".

Recorded, not fixed here (PLAN §6)

An unknown query operator matches nothing instead of erroring.
{x: {$bogus: 1}} answers ok: 1 with an empty result on find, count
and aggregate alike, where mongod answers BadValue on all three.
Measured on both servers side by side. Same class as M2's six silent wrong
answers, and it lives in the shared query path -- one fix for every command.
A non-string collection name is dispatch's BadValue where mongod says
InvalidNamespace (73): one answer for the whole command table.
distinct is unbounded in memory, joining $group and $sort.

Verification

198/198 unit (7 new) in ReleaseFast and ReleaseSafe, 83/83 fuzz, aggregation
corpus 70/0, full e2e matrix and crash-fuzz green.

M3 opens with the cheapest lever: a whole command that did not exist. Five corpus cases answered "no such command". Two things were measured against mongod 8.3.7 rather than recalled, and the first is not what anyone would guess: - The answer is **sorted in canonical BSON order**, not in the order the values were met. `{s: "b"}, {s: "a"}, {s: null}` answers `[null, "a", "b"]`. Insertion order is the obvious implementation, it passes every test anybody would think to write by hand against `[11, 22, 33]`, and it is wrong. - Deduping is the same comparator, so an int32 `1` and a double `1.0` collapse while `null` and `"1"` survive. Both fall out of `bson.compare`, which `$sort` and `$min` already use -- and that is not luck: mongod accumulates into a `BSONElementSet` ordered by the same `woCompare`. The rest reuses the shared read path. Running the identical probe against both servers now agrees on **every semantic row**. ## Scorecard 201 pass / 90 fail -> **204 / 87**. `distinct.json` 0/2 -> 2/0, `distinct-rawdata` 0/1 -> 1/0. `distinct-comment` nets zero: its "no such command" is replaced by the pre-4.4.14 document-comment case, which `estimatedDocumentCount` already carries as a standing failure -- emulating a bug fixed in 4.4.14 for one command would make the two disagree. `distinct-collation` still needs M8, but now fails with the honest "expected 1 elements, got 2". ## Recorded, not fixed here (PLAN §6) - **An unknown query operator matches nothing instead of erroring.** `{x: {$bogus: 1}}` answers `ok: 1` with an empty result on `find`, `count` and `aggregate` alike, where mongod answers `BadValue` on all three. Measured on both servers side by side. Same class as M2's six silent wrong answers, and it lives in the shared query path -- one fix for every command. - A non-string collection name is dispatch's `BadValue` where mongod says `InvalidNamespace` (73): one answer for the whole command table. - `distinct` is unbounded in memory, joining `$group` and `$sort`. ## Verification 198/198 unit (7 new) in ReleaseFast and ReleaseSafe, 83/83 fuzz, aggregation corpus 70/0, full e2e matrix and crash-fuzz green.

dev added 2 commits 2026-08-10 14:42:21 +00:00

commands: distinct 1f141ef619

A whole command that did not exist: five corpus cases answered "no such
command". Two things about it were measured against mongod 8.3.7 rather
than recalled, and the first is not what anyone would guess.

  - The answer is **sorted in canonical BSON order**, not in the order the
    values were met. `{s: "b"}, {s: "a"}, {s: null}` answers
    `[null, "a", "b"]`. Insertion order is the obvious implementation, it
    passes every test anybody would think to write by hand against
    `[11, 22, 33]`, and it is wrong.
  - Deduping is the same comparator, so an int32 `1` and a double `1.0`
    collapse while `null` and `"1"` survive.

Both fall out of `bson.compare`, which `$sort` and `$min` already use --
and that is not luck: mongod accumulates into a `BSONElementSet` ordered by
the same `woCompare`. The rest reuses the shared read path: byte-walked
`collect_values_bytes` for the key, so the traversal, the multikey descent
and the numeric path segments are the ones the matcher and the index
already agree on.

Also measured: a terminal array contributes its elements exactly one level
deep (`[[7, 8], 9]` gives `[7, 8]` and `9`, never 7 and 8); a missing field
contributes nothing where an explicit null contributes null; an absent
collection, an absent database and an empty key are each `ok: 1` with an
empty array rather than an error; `query` absent and `query: null` are both
an empty filter; a missing `key` is IDLFailedToParse (40414) while a
wrong-typed one is TypeMismatch (14).

Running the identical probe against both servers now agrees on every
semantic row. Three divergences remain, all outside this command and
recorded in PLAN §6: an unknown query operator matches nothing instead of
erroring (shared with find/count/aggregate, and the same class as M2's six
silent wrong answers), a non-string collection name is refused by dispatch
as BadValue where mongod says InvalidNamespace, and an unknown top-level
field is tolerated -- deliberately, since `comment` and `rawData` arrive
through that door and the corpus requires both be ignored.

crud scorecard: 201 pass / 90 fail -> 204 / 87. distinct.json 0/2 -> 2/0,
distinct-rawdata 0/1 -> 1/0. distinct-comment nets zero: its "no such
command" is replaced by the pre-4.4.14 document-comment case, which
`estimatedDocumentCount` already carries as a standing failure -- and
emulating a bug fixed in 4.4.14 for one command would make the two
disagree. distinct-collation still needs M8, but now fails with the honest
"expected 1 elements, got 2".

198/198 unit (7 new) in ReleaseFast and ReleaseSafe, 83/83 fuzz, aggregation
corpus 70/0, full e2e matrix and crash-fuzz green.

plan: what measuring distinct turned up 5942f5e65f

M3 opens with `distinct` recorded as done and `arrayFilters` named in its
scope, and §6 gains an M3 entry for the three findings the measurement
produced that are not `distinct`'s to fix.

The one worth reading twice: an unknown query operator answers `ok: 1` with
an empty result on find, count and aggregate alike, where mongod answers
BadValue on all three. Measured on both servers side by side. That is the
same shape as M2's six silent wrong answers -- a typo'd operator reads as
"no matches" -- and it sits in the shared query path, so it is one fix for
every command rather than one per command.

Also recorded: the InvalidNamespace divergence is dispatch's answer for the
whole command table, not distinct's; and distinct joins $group and $sort on
the list of things unbounded in memory.

dev merged commit 5942f5e65f into main

2026-08-10 14:42:40 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dev/MultiforaDB#4