M3: the positional operators and arrayFilters #7

Merged
dev merged 4 commits from m3-positional into main 2026-08-10 17:49:33 +00:00
6 changed files with 1325 additions and 225 deletions

33
PLAN.md
View File

@@ -467,7 +467,7 @@ answers, and the trade is only acceptable because the lie is removed first.
| M1 | **Cursors + wire polish** | getMore / killCursors / batchSize; server-side cursor state with idle timeout; sessions plumbing (lsid accepted) as drivers send it; hello advertisement updates; **`moreToCome` on requests** (see the bug below); command-monitoring assertions in the spec runner | crud spec suite green; e2e green | | M1 | **Cursors + wire polish** | getMore / killCursors / batchSize; server-side cursor state with idle timeout; sessions plumbing (lsid accepted) as drivers send it; hello advertisement updates; **`moreToCome` on requests** (see the bug below); command-monitoring assertions in the spec runner | crud spec suite green; e2e green |
| M2 | **The `aggregate` command surface** | `$out` and `$merge` (7 of the 13 failures), and refusing every pipeline construct the engine does not implement instead of answering `0` (amendment A6). The other 6 failures are blocked on M2.5, M4 and M8 — see `docs/M2_DESIGN_REVIEW.md` §7 | `aggregate-*.json`: 0 fail among the 7 reachable cases | | M2 | **The `aggregate` command surface** | `$out` and `$merge` (7 of the 13 failures), and refusing every pipeline construct the engine does not implement instead of answering `0` (amendment A6). The other 6 failures are blocked on M2.5, M4 and M8 — see `docs/M2_DESIGN_REVIEW.md` §7 | `aggregate-*.json`: 0 fail among the 7 reachable cases |
| M2.5 | **The aggregation engine** | expression evaluator, per-stage document iterator, the accumulators, `$unwind`; `$lookup`/`$facet` explicitly out of the first cut (amendment A6) | a purpose-built stage corpus, every expectation measured against mongod | | M2.5 | **The aggregation engine** | expression evaluator, per-stage document iterator, the accumulators, `$unwind`; `$lookup`/`$facet` explicitly out of the first cut (amendment A6) | a purpose-built stage corpus, every expectation measured against mongod |
| M3 | **Update operators + index types** | `distinct` (**done**); positional paths refused rather than destructive (**done**); then `$`/`$[]`/`$[<ident>]` implemented, $setOnInsert, $addToSet, $mul, $min/$max, $pop, $pullAll, $currentDate, pipeline updates; partial + hashed indexes | `tests/spec/positional/` 0 fail (51 cases, recorded from mongod — the named gate could not see this work, see below); remaining crud coverage; e2e3/e2e4 green | | M3 | **Update operators + index types** | `distinct` (**done**); positional paths refused rather than destructive (**done**); `$`/`$[]`/`$[<ident>]` + `arrayFilters` implemented (**done**); then $setOnInsert, $addToSet, $mul, $min/$max, $pop, $pullAll, $currentDate, pipeline updates; partial + hashed indexes | `tests/spec/positional/` 0 fail (51 cases, recorded from mongod — **green**; the named gate could not see this work, see below); remaining crud coverage; e2e3/e2e4 green |
| M4 | **Sessions + transactions** | logical sessions, snapshot isolation on the mmap engine, write concern at commit | sessions + transactions spec suites green | | M4 | **Sessions + transactions** | logical sessions, snapshot isolation on the mmap engine, write concern at commit | sessions + transactions spec suites green |
| M5 | **Change streams** | change feed + resume tokens (likely log-seq based), getMore integration | change-streams spec suite green | | M5 | **Change streams** | change feed + resume tokens (likely log-seq based), getMore integration | change-streams spec suite green |
| M6 | **Admin/ops commands** | dbStats, collStats, serverStatus, ping, buildInfo, listDatabases filters, dropDatabase durability (log it) | mongosh UX smoke; e2e green | | M6 | **Admin/ops commands** | dbStats, collStats, serverStatus, ping, buildInfo, listDatabases filters, dropDatabase durability (log it) | mongosh UX smoke; e2e green |
@@ -586,9 +586,10 @@ gaps (`bad update`, `update must be a document` — M3), unimplemented commands
`bulkWrite`/`insertMany`. That list, not the total, is the milestone backlog. `bulkWrite`/`insertMany`. That list, not the total, is the milestone backlog.
All three named commands have since landed: `$out`/`$merge` in M2, `distinct` All three named commands have since landed: `$out`/`$merge` in M2, `distinct`
as M3's first commit. The backlog the same grouping gives today is as M3's first commit, and the `arrayFilters` cluster (14 cases) with M3's
`arrayFilters` (14 cases), the `findOneAndUpdate`/`findOneAndReplace` shapes positional operators. The backlog the same grouping gives today is the
(~10), pipeline-form updates (~10), and `create-null-ids` (6). `findOneAndUpdate`/`findOneAndReplace` shapes (~10), pipeline-form updates
(~10), and `create-null-ids` (6).
--- ---
@@ -1091,7 +1092,7 @@ has to be its own commit with its own re-recorded scorecard.
`sources/`, every expectation recorded from mongod 8.3.7, run through the `sources/`, every expectation recorded from mongod 8.3.7, run through the
shared runner with `--suite-dir`. shared runner with `--suite-dir`.
It stands at 15 pass / 36 fail against the refusal, which is the intended It stood at 15 pass / 36 fail against the refusal, which was the intended
shape — `expressions.json` was 1/26 before the evaluator and is green now. shape — `expressions.json` was 1/26 before the evaluator and is green now.
Recording it settled a dozen things, and the first contradicts what the Recording it settled a dozen things, and the first contradicts what the
design review assumed: `y.$[i].c.$[i].d` reusing one identifier at two design review assumed: `y.$[i].c.$[i].d` reusing one identifier at two
@@ -1100,6 +1101,28 @@ has to be its own commit with its own re-recorded scorecard.
is error 28 where every other path failure is 2; a positional segment never is error 28 where every other path failure is 2; a positional segment never
creates, so a missing or non-array path is an error where `$set: {'a.b': 1}` creates, so a missing or non-array path is an error where `$set: {'a.b': 1}`
would construct; and an upsert gets no special case. would construct; and an upsert gets no special case.
**Green as of the two implementation commits**, 51/51, and the pinned crud
scorecard moved 204 → 218 with it. Three divergences from mongod were
measured on the way and left in place rather than guessed at:
- **`$` with two predicates on one array.** `{"y.b": 3, "y.c": 2}` matches
`[{b: 3, c: 1}, {b: 1, c: 2}]` without either element satisfying both, and
the document still matched. mongod writes element 1; this writes element 0.
mongod's answer is an artefact of which predicate last wrote its match
position — the same query with the two predicates *reversed* still gives 1
— so there is no rule here to copy, only a behaviour to record.
- **An array filter with a top-level `$and`/`$or`.** mongod accepts
`[{$or: [{"i.b": 3}]}]` and finds the identifier inside it; this refuses
with 9, the same answer it gives `[{}]`. The identifier is read off
top-level field names, and an operator is not one.
- **A literal index into a scalar element.** `$set: {"y.0.b": 1}` on
`y: [null]` is PathNotViable (28) on mongod; here it replaces the null
with `{b: 1}`. The positional walk refuses this — `Walk.element` is where
the corpus measures it — but the plain indexed path still takes
`set_path`'s creating branch. One rule, reached two ways, and only one way
agrees. Worth one commit, and it needs its own measurements first: the
padding case (`y.3.b` past the end) is a *legal* creation on mongod, so
the fix is not "refuse a non-document element".
- **M3 update operators** — open. `distinct` landed first because it was a - **M3 update operators** — open. `distinct` landed first because it was a
whole missing command with no dependencies, and measuring it turned up three whole missing command with no dependencies, and measuring it turned up three
things worth keeping, none of which are `distinct`'s to fix: things worth keeping, none of which are `distinct`'s to fix:

View File

@@ -10,6 +10,14 @@ Everything below was measured on 2026-08-10 against mongod 8.3.7 on
`:27099` and this server at `5942f5e` on `:27020`, running the identical `:27099` and this server at `5942f5e` on `:27020`, running the identical
probe against both. probe against both.
**Outcome.** All of it landed, in the order §6 recommended: the refusal
(`f04e712`), the corpus (`e5a84c0`), then the implementation in two commits.
The corpus is 51/51 and the pinned crud scorecard moved 204 → 218. Two of the
review's own guesses were wrong and the corpus caught both — a reused
identifier is accepted, and `$[]` in first position answers the array-filter
message rather than the `$` one. The divergences that remain are listed in
PLAN §6 under the M3 gate; this document is not the place to track them.
--- ---
## 1. The plan names a feature; the measurement found data loss ## 1. The plan names a feature; the measurement found data loss

View File

@@ -1969,15 +1969,30 @@ fn cmd_update(ctx: *Context, msg: *wire.Message, reply: *wire.Reply) !void {
return failed_to_parse(reply, "multi update is not supported for replacement-style update"); return failed_to_parse(reply, "multi update is not supported for replacement-style update");
} }
var diag: update.Diagnostic = .{};
const opts = update.Options{
.array_filters = try parse_array_filters(
reply,
spec.get("arrayFilters"),
"update.updates.arrayFilters",
) orelse return,
.query = q,
.diag = &diag,
};
// Before the scan, not after: an update naming an identifier nothing
// binds is refused whether or not it would have matched anything, and
// an array filter the update never uses is refused even when the whole
// command was a no-op. Both measured.
update.validate(u_doc, opts) catch |err| return update_refusal(reply, err, diag);
var matched: std.ArrayListUnmanaged(u64) = .empty; var matched: std.ArrayListUnmanaged(u64) = .empty;
defer matched.deinit(ctx.gpa); defer matched.deinit(ctx.gpa);
_ = try scan_matching(ctx, db_name, coll_name, q, if (multi) 0 else 1, &matched); _ = try scan_matching(ctx, db_name, coll_name, q, if (multi) 0 else 1, &matched);
if (matched.items.len == 0) { if (matched.items.len == 0) {
if (upsert) { if (upsert) {
var up_diag: update.Diagnostic = .{}; const new_doc = build_upsert_doc(reply, q, u_doc, opts) catch |err|
const new_doc = build_upsert_doc(reply, q, u_doc, &up_diag) catch |err| return update_refusal(reply, err, diag);
return update_refusal(reply, err, up_diag);
ctx.engine.insert(db_name, coll_name, new_doc, ctx.oid_gen) catch |err| switch (err) { ctx.engine.insert(db_name, coll_name, new_doc, ctx.oid_gen) catch |err| switch (err) {
error.DuplicateKey, error.DuplicateKeyIndex => return duplicate_key_error(ctx, reply, db_name, coll_name, new_doc), error.DuplicateKey, error.DuplicateKeyIndex => return duplicate_key_error(ctx, reply, db_name, coll_name, new_doc),
else => return err, else => return err,
@@ -1999,8 +2014,7 @@ fn cmd_update(ctx: *Context, msg: *wire.Message, reply: *wire.Reply) !void {
// and a rejected update must not corrupt the stored document. // and a rejected update must not corrupt the stored document.
const doc = try doc_tree(reply.arena_alloc(), coll, off); const doc = try doc_tree(reply.arena_alloc(), coll, off);
const copy = try clone_doc(reply, doc); const copy = try clone_doc(reply, doc);
var diag: update.Diagnostic = .{}; update.apply(copy, &.{ .arena = undefined, .pairs = u_doc }, opts) catch |err|
update.apply(copy, &.{ .arena = undefined, .pairs = u_doc }, &diag) catch |err|
return update_refusal(reply, err, diag); return update_refusal(reply, err, diag);
const written = ctx.engine.replace(db_name, coll_name, copy, ctx.oid_gen) catch |err| switch (err) { const written = ctx.engine.replace(db_name, coll_name, copy, ctx.oid_gen) catch |err| switch (err) {
error.DuplicateKey, error.DuplicateKeyIndex => { error.DuplicateKey, error.DuplicateKeyIndex => {
@@ -2080,6 +2094,20 @@ fn cmd_find_and_modify(ctx: *Context, msg: *wire.Message, reply: *wire.Reply) !v
if (remove and do_update) return bad_value(reply, "remove and update are mutually exclusive"); if (remove and do_update) return bad_value(reply, "remove and update are mutually exclusive");
if (!remove and !do_update) return bad_value(reply, "must specify update or remove"); if (!remove and !do_update) return bad_value(reply, "must specify update or remove");
var diag: update.Diagnostic = .{};
const opts = update.Options{
.array_filters = try parse_array_filters(
reply,
msg.body.get("arrayFilters"),
"findAndModify.arrayFilters",
) orelse return,
.query = q,
.diag = &diag,
};
if (doc_arg(msg.body.get("update"))) |u| {
update.validate(u, opts) catch |err| return update_refusal(reply, err, diag);
}
var matched: std.ArrayListUnmanaged(u64) = .empty; var matched: std.ArrayListUnmanaged(u64) = .empty;
defer matched.deinit(ctx.gpa); defer matched.deinit(ctx.gpa);
// Without a sort, only the first match is ever used. // Without a sort, only the first match is ever used.
@@ -2104,9 +2132,8 @@ fn cmd_find_and_modify(ctx: *Context, msg: *wire.Message, reply: *wire.Reply) !v
if (target == null and do_update and upsert) { if (target == null and do_update and upsert) {
const u_doc = doc_arg(msg.body.get("update")) orelse return bad_value(reply, "update must be a document"); const u_doc = doc_arg(msg.body.get("update")) orelse return bad_value(reply, "update must be a document");
var up_diag: update.Diagnostic = .{}; const new_doc = build_upsert_doc(reply, q, u_doc, opts) catch |err|
const new_doc = build_upsert_doc(reply, q, u_doc, &up_diag) catch |err| return update_refusal(reply, err, diag);
return update_refusal(reply, err, up_diag);
ctx.engine.insert(db_name, coll_name, new_doc, ctx.oid_gen) catch |err| switch (err) { ctx.engine.insert(db_name, coll_name, new_doc, ctx.oid_gen) catch |err| switch (err) {
error.DuplicateKey, error.DuplicateKeyIndex => return duplicate_key_error(ctx, reply, db_name, coll_name, new_doc), error.DuplicateKey, error.DuplicateKeyIndex => return duplicate_key_error(ctx, reply, db_name, coll_name, new_doc),
else => return err, else => return err,
@@ -2123,8 +2150,7 @@ fn cmd_find_and_modify(ctx: *Context, msg: *wire.Message, reply: *wire.Reply) !v
const u_doc = doc_arg(msg.body.get("update")) orelse return bad_value(reply, "update must be a document"); const u_doc = doc_arg(msg.body.get("update")) orelse return bad_value(reply, "update must be a document");
const before = try bson.copy_pairs(arena, target.?.pairs); const before = try bson.copy_pairs(arena, target.?.pairs);
const copy = try clone_doc(reply, target.?); const copy = try clone_doc(reply, target.?);
var diag: update.Diagnostic = .{}; update.apply(copy, &.{ .arena = undefined, .pairs = u_doc }, opts) catch |err|
update.apply(copy, &.{ .arena = undefined, .pairs = u_doc }, &diag) catch |err|
return update_refusal(reply, err, diag); return update_refusal(reply, err, diag);
// findAndModify reports `n` (matched) and `updatedExisting`, neither of // findAndModify reports `n` (matched) and `updatedExisting`, neither of
// which distinguishes a no-op, so whether it wrote is not needed here. // which distinguishes a no-op, so whether it wrote is not needed here.
@@ -4141,20 +4167,92 @@ fn clone_doc(reply: *wire.Reply, doc: *const bson.Document) !*bson.Document {
/// `findAndModify` and the upsert path -- and the two new ones exist to stop /// `findAndModify` and the upsert path -- and the two new ones exist to stop
/// a silent divergence in the first place. /// a silent divergence in the first place.
/// ///
/// The messages are this server's own words. mongod's `PathNotViable` text /// Every message here was measured on mongod 8.3.7 and is reproduced verbatim,
/// embeds a shell-syntax rendering of the offending element (`Cannot create /// except where mongod's text embeds a shell-syntax rendering of the offending
/// field 'nope' in element {y: [ { b: 3 }, { b: 1 } ]}`), and there is no BSON /// BSON (`Cannot create field 'nope' in element {y: [ { b: 3 } ]}`): there is
/// formatter here that produces it. The code is what the corpus asserts and /// no formatter here that produces it, and a half-copy would be worse than a
/// the code is exact; a half-copy of the text would be worse than a clear /// clear sentence that does not pretend. Codes are exact throughout, and codes
/// sentence that does not pretend. /// are what the corpus asserts.
///
/// mongod wraps the refusals it only reaches with a document in hand in
/// `Plan executor error during update :: caused by :: `. That prefix is
/// dropped here: it names a mongod component this server does not have.
fn update_refusal(reply: *wire.Reply, err: anyerror, diag: update.Diagnostic) !void { fn update_refusal(reply: *wire.Reply, err: anyerror, diag: update.Diagnostic) !void {
const arena = reply.arena_alloc(); const arena = reply.arena_alloc();
switch (err) { switch (err) {
error.PositionalUnsupported => return bad_value(reply, try std.fmt.allocPrint( error.PositionalFirst => return bad_value(reply, if (std.mem.eql(u8, diag.segment, "$"))
try std.fmt.allocPrint(
arena, arena,
"the positional operator '{s}' in path '{s}' is not implemented by this server", "Cannot have positional (i.e. '$') element in the first position in path '{s}'",
.{diag.path},
)
else
try std.fmt.allocPrint(
arena,
"Cannot have array filter identifier (i.e. '$[<id>]') element in the " ++
"first position in path '{s}'",
.{diag.path},
)),
error.TooManyPositional => return bad_value(reply, try std.fmt.allocPrint(
arena,
"Too many positional (i.e. '$') elements found in path '{s}'",
.{diag.path},
)),
error.NoArrayFilter => return bad_value(reply, try std.fmt.allocPrint(
arena,
"No array filter found for identifier '{s}' in path '{s}'",
.{ diag.segment, diag.path }, .{ diag.segment, diag.path },
)), )),
error.BadArrayFilterIdentifier => return bad_value(reply, try std.fmt.allocPrint(
arena,
"Error parsing array filter :: caused by :: The top-level field name must be an " ++
"alphanumeric string beginning with a lowercase letter, found '{s}'",
.{diag.segment},
)),
error.NoPositionalMatch => return bad_value(
reply,
"The positional operator did not find the match needed from the query.",
),
error.ArrayPathRequired => return bad_value(reply, try std.fmt.allocPrint(
arena,
"The path '{s}' must exist in the document in order to apply array updates.",
.{diag.path},
)),
error.NotAnArrayPath => return bad_value(reply, try std.fmt.allocPrint(
arena,
"Cannot apply array updates to non-array element at path '{s}'",
.{diag.path},
)),
error.RenameDynamicSource => return bad_value(reply, try std.fmt.allocPrint(
arena,
"The source field for $rename may not be dynamic: {s}",
.{diag.path},
)),
error.RenameDynamicDestination => return bad_value(reply, try std.fmt.allocPrint(
arena,
"The destination field for $rename may not be dynamic: {s}",
.{diag.path},
)),
error.UnusedArrayFilter => return failed_to_parse(reply, try std.fmt.allocPrint(
arena,
"The array filter for identifier '{s}' was not used in the update",
.{diag.segment},
)),
error.DuplicateArrayFilter => return failed_to_parse(reply, try std.fmt.allocPrint(
arena,
"Found multiple array filters with the same top-level field name {s}",
.{diag.segment},
)),
error.EmptyArrayFilter => return failed_to_parse(
reply,
"Cannot use an expression without a top-level field name in arrayFilters",
),
error.MultipleArrayFilterIdentifiers => return failed_to_parse(reply, try std.fmt.allocPrint(
arena,
"Error parsing array filter :: caused by :: Expected a single top-level field " ++
"name, found '{s}' and '{s}'",
.{ diag.segment, diag.other },
)),
error.PathNotViable => return reply.put_error( error.PathNotViable => return reply.put_error(
@intFromEnum(ErrorCode.path_not_viable), @intFromEnum(ErrorCode.path_not_viable),
"PathNotViable", "PathNotViable",
@@ -4169,13 +4267,75 @@ fn update_refusal(reply: *wire.Reply, err: anyerror, diag: update.Diagnostic) !v
} }
} }
/// Read an `arrayFilters` argument into the bindings `$[<identifier>]`
/// resolves against.
///
/// Only the shape is the command's business, and only the shape is checked
/// here: which identifier a filter names, whether it is spelled legally and
/// whether the update ever uses it all belong to `update.validate`, which is
/// the half that can see the update's paths. `field` is the dotted name
/// mongod puts in the message, and it differs between the two callers.
///
/// Returns null having written the error reply, like the other `*_arg`
/// helpers.
fn parse_array_filters(
reply: *wire.Reply,
value: ?bson.Value,
comptime field: []const u8,
) !?[]update.ArrayFilter {
const v = value orelse return &.{};
const arr = switch (v) {
.array => |a| a,
else => {
try wrong_type(reply, field, v, "array");
return null;
},
};
const out = try reply.arena_alloc().alloc(update.ArrayFilter, arr.len);
for (arr, 0..) |elem, i| {
out[i] = .{ .pairs = switch (elem) {
.doc => |d| d,
else => {
const arena = reply.arena_alloc();
const name = try std.fmt.allocPrint(arena, field ++ ".{d}", .{i});
try wrong_type_dynamic(reply, name, elem, "object");
return null;
},
} };
}
return out;
}
fn wrong_type(
reply: *wire.Reply,
comptime field: []const u8,
got: bson.Value,
comptime want: []const u8,
) !void {
return wrong_type_dynamic(reply, field, got, want);
}
fn wrong_type_dynamic(
reply: *wire.Reply,
field: []const u8,
got: bson.Value,
want: []const u8,
) !void {
const text = try std.fmt.allocPrint(
reply.arena_alloc(),
"BSON field '{s}' is the wrong type '{s}', expected type '{s}'",
.{ field, got.type_name(), want },
);
return reply.put_error(@intFromEnum(ErrorCode.type_mismatch), "TypeMismatch", text);
}
/// Build the document for an upsert: equality fields from the filter, then /// Build the document for an upsert: equality fields from the filter, then
/// the update operators applied. Owned by the reply arena. /// the update operators applied. Owned by the reply arena.
fn build_upsert_doc( fn build_upsert_doc(
reply: *wire.Reply, reply: *wire.Reply,
q: []const bson.Pair, q: []const bson.Pair,
u_doc: []const bson.Pair, u_doc: []const bson.Pair,
diag: *update.Diagnostic, opts: update.Options,
) !*bson.Document { ) !*bson.Document {
const arena = reply.arena_alloc(); const arena = reply.arena_alloc();
var pairs: std.ArrayListUnmanaged(bson.Pair) = .empty; var pairs: std.ArrayListUnmanaged(bson.Pair) = .empty;
@@ -4190,7 +4350,7 @@ fn build_upsert_doc(
const owned = try arena.create(bson.Document); const owned = try arena.create(bson.Document);
owned.* = bson.Document{ .arena = std.heap.ArenaAllocator.init(arena), .pairs = try pairs.toOwnedSlice(arena) }; owned.* = bson.Document{ .arena = std.heap.ArenaAllocator.init(arena), .pairs = try pairs.toOwnedSlice(arena) };
// Apply update operators to build the final doc; _id handled by insert. // Apply update operators to build the final doc; _id handled by insert.
try update.apply(owned, &.{ .arena = undefined, .pairs = u_doc }, diag); try update.apply(owned, &.{ .arena = undefined, .pairs = u_doc }, opts);
return owned; return owned;
} }
@@ -5629,9 +5789,9 @@ test "a positional update is refused on the wire and stores nothing" {
var ctx = tdb.ctx(io); var ctx = tdb.ctx(io);
const cases = [_]struct { coll: []const u8, path: []const u8, code: i32 }{ const cases = [_]struct { coll: []const u8, path: []const u8, code: i32 }{
.{ .coll = "a1", .path = "y.$[i].b", .code = 2 }, // filtered positional .{ .coll = "a1", .path = "y.$[i].b", .code = 2 }, // no array filter binds `i`
.{ .coll = "a2", .path = "y.$[].b", .code = 2 }, // all-positional .{ .coll = "a2", .path = "$[]", .code = 2 }, // positional in first position
.{ .coll = "a3", .path = "y.$.b", .code = 2 }, // positional .{ .coll = "a3", .path = "y.$.b", .code = 2 }, // no predicate for `$` to use
.{ .coll = "a4", .path = "y.nope.b", .code = 28 }, // PathNotViable, same branch .{ .coll = "a4", .path = "y.nope.b", .code = 28 }, // PathNotViable, same branch
}; };
for (cases) |c| { for (cases) |c| {
@@ -5669,6 +5829,134 @@ test "a positional update is refused on the wire and stores nothing" {
} }
} }
test "an all-positional update writes every element on the wire" {
// The other end of the same chain: `$[]` reaches the stored document, and
// reaches *all* of it. `distinct` on `y.b` is the check that says so in one
// number -- two values means only one element moved.
var threaded: std.Io.Threaded = .init_single_threaded;
defer threaded.deinit();
const io = threaded.io();
var tdb = try TestDb.init(io);
defer tdb.deinit();
var ctx = tdb.ctx(io);
try dispatch_insert(&tdb, io, "all", &.{
.{ .doc = &.{
.{ .key = "_id", .value = .{ .int32 = 1 } },
.{ .key = "y", .value = .{ .array = &.{
.{ .doc = &.{.{ .key = "b", .value = .{ .int32 = 3 } }} },
.{ .doc = &.{.{ .key = "b", .value = .{ .int32 = 1 } }} },
} } },
} },
});
const updates = [_]bson.Value{.{ .doc = &.{
.{ .key = "q", .value = .{ .doc = &.{} } },
.{ .key = "u", .value = .{ .doc = &.{
.{ .key = "$set", .value = .{ .doc = &.{
.{ .key = "y.$[].b", .value = .{ .int32 = 9 } },
} } },
} } },
} }};
try testing.expectEqual(@as(?i32, null), try run_for_code(&ctx, "update", .{ .string = "all" }, &.{
.{ .key = "updates", .value = .{ .array = &updates } },
}));
var reply = wire.Reply.init(testing.allocator);
defer reply.deinit();
const values = try distinct_values(&tdb, io, &reply, "all", &.{
.{ .key = "key", .value = .{ .string = "y.b" } },
});
try testing.expectEqual(@as(usize, 1), values.len);
try testing.expectEqual(@as(i32, 9), values[0].int32);
}
test "arrayFilters reach the update, and are refused before the scan" {
// The plumbing, end to end: an identifier in the path only means anything
// if the filter beside it arrives with it.
var threaded: std.Io.Threaded = .init_single_threaded;
defer threaded.deinit();
const io = threaded.io();
var tdb = try TestDb.init(io);
defer tdb.deinit();
var ctx = tdb.ctx(io);
try dispatch_insert(&tdb, io, "af", &.{
.{ .doc = &.{
.{ .key = "_id", .value = .{ .int32 = 1 } },
.{ .key = "y", .value = .{ .array = &.{
.{ .doc = &.{.{ .key = "b", .value = .{ .int32 = 3 } }} },
.{ .doc = &.{.{ .key = "b", .value = .{ .int32 = 1 } }} },
} } },
} },
});
const filters = [_]bson.Value{.{ .doc = &.{.{ .key = "i.b", .value = .{ .int32 = 3 } }} }};
const updates = [_]bson.Value{.{ .doc = &.{
.{ .key = "q", .value = .{ .doc = &.{} } },
.{ .key = "u", .value = .{ .doc = &.{
.{ .key = "$set", .value = .{ .doc = &.{
.{ .key = "y.$[i].b", .value = .{ .int32 = 9 } },
} } },
} } },
.{ .key = "arrayFilters", .value = .{ .array = &filters } },
} }};
try testing.expectEqual(@as(?i32, null), try run_for_code(&ctx, "update", .{ .string = "af" }, &.{
.{ .key = "updates", .value = .{ .array = &updates } },
}));
// Only the element the filter selected moved. Mutation check: drop the
// `arrayFilters` read in `cmd_update` and this is a `NoArrayFilter` reply
// instead, which is the honest failure -- but silently ignoring the field
// would write both elements.
var reply = wire.Reply.init(testing.allocator);
defer reply.deinit();
const values = try distinct_values(&tdb, io, &reply, "af", &.{
.{ .key = "key", .value = .{ .string = "y.b" } },
});
try testing.expectEqual(@as(usize, 2), values.len);
try testing.expectEqual(@as(i32, 1), values[0].int32);
try testing.expectEqual(@as(i32, 9), values[1].int32);
// A filter no path uses is refused even when the query matches nothing at
// all, which is what makes the check belong beside the scan rather than
// inside `apply`. Mutation check: delete the standalone `update.validate`
// call and this answers ok -- `apply` runs once per matched document, and
// there are none.
const unused = [_]bson.Value{.{ .doc = &.{
.{ .key = "q", .value = .{ .doc = &.{.{ .key = "nomatch", .value = .{ .int32 = 1 } }} } },
.{ .key = "u", .value = .{ .doc = &.{
.{ .key = "$set", .value = .{ .doc = &.{.{ .key = "y.b", .value = .{ .int32 = 2 } }} } },
} } },
.{ .key = "arrayFilters", .value = .{ .array = &filters } },
} }};
try testing.expectEqual(@as(?i32, 9), try run_for_code(&ctx, "update", .{ .string = "af" }, &.{
.{ .key = "updates", .value = .{ .array = &unused } },
}));
// Shape is the command's business, and it answers TypeMismatch for it.
const bad = [_]bson.Value{.{ .doc = &.{
.{ .key = "q", .value = .{ .doc = &.{} } },
.{ .key = "u", .value = .{ .doc = &.{
.{ .key = "$set", .value = .{ .doc = &.{.{ .key = "y.$[i].b", .value = .{ .int32 = 9 } }} } },
} } },
.{ .key = "arrayFilters", .value = .{ .array = &.{.{ .int32 = 3 }} } },
} }};
try testing.expectEqual(@as(?i32, 14), try run_for_code(&ctx, "update", .{ .string = "af" }, &.{
.{ .key = "updates", .value = .{ .array = &bad } },
}));
const not_an_array = [_]bson.Value{.{ .doc = &.{
.{ .key = "q", .value = .{ .doc = &.{} } },
.{ .key = "u", .value = .{ .doc = &.{
.{ .key = "$set", .value = .{ .doc = &.{.{ .key = "y.$[i].b", .value = .{ .int32 = 9 } }} } },
} } },
.{ .key = "arrayFilters", .value = .{ .int32 = 1 } },
} }};
try testing.expectEqual(@as(?i32, 14), try run_for_code(&ctx, "update", .{ .string = "af" }, &.{
.{ .key = "updates", .value = .{ .array = &not_an_array } },
}));
}
test "aggregate $sort without a preceding $group sorts and frees correctly" { test "aggregate $sort without a preceding $group sorts and frees correctly" {
// Regression test for a remote, client-triggerable invalid free: the // Regression test for a remote, client-triggerable invalid free: the
// $sort stage materialized its document list from the reply arena and // $sort stage materialized its document list from the reply arena and

File diff suppressed because it is too large Load Diff

View File

@@ -72,24 +72,26 @@ measures the version gap, not the engine.
## Where it stands ## Where it stands
Recorded against mongod 8.3.7, run against the server at the positional Recorded against mongod 8.3.7. Green:
refusal:
``` ```
filtered.json 8 pass 19 fail 0 skip filtered.json 27 pass 0 fail 0 skip
all-positional.json 3 pass 9 fail 0 skip all-positional.json 12 pass 0 fail 0 skip
first-positional.json 4 pass 8 fail 0 skip first-positional.json 12 pass 0 fail 0 skip
``` ```
Red by construction and by design. The 15 that pass are the refusals where It was recorded red — 15 pass / 36 fail against the positional refusal — and
this server's code already matches mongod's; of the 36 that fail, 31 are the driven green by the two implementation commits, the same shape
constructs themselves answering "not implemented by this server", and 5 are `tests/spec/aggregate/expressions.json` had at 1 pass / 26 fail before the
refusals whose code differs — four of them the `arrayFilters` validation this expression evaluator existed. The 15 that passed then were the refusals where
server cannot do yet because it never parses the option. this server already agreed with mongod, which is the only part of a red gate
that is worth anything: it says the corpus is measuring the server and not
the harness.
That is the intended shape. `tests/spec/aggregate/expressions.json` was Three answers here still differ from mongod and are green only because no
recorded at 1 pass / 26 fail before the evaluator existed and is green now; case covers them; they are written down in PLAN §6 rather than papered over —
this is the same gate at the same stage. `$` with two disagreeing predicates on one array, an array filter with a
top-level `$and`/`$or`, and a literal index into a scalar element.
## What recording it settled ## What recording it settled

View File

@@ -18,7 +18,7 @@
# hasServerConnectionId, and maxTimeMS in an expected command (CSOT rewrites # hasServerConnectionId, and maxTimeMS in an expected command (CSOT rewrites
# it -- the only assertion this runner declines to make). # it -- the only assertion this runner declines to make).
total 204 pass 87 fail 196 skip 175 files 0 errored total 218 pass 73 fail 196 skip 175 files 0 errored
# per-file: name pass fail skip # per-file: name pass fail skip
aggregate-allowdiskuse.json 3 0 0 aggregate-allowdiskuse.json 3 0 0
@@ -31,7 +31,7 @@ aggregate-out.json 2 0 0
aggregate-rawdata.json 1 0 1 aggregate-rawdata.json 1 0 1
aggregate-write-readPreference.json 0 0 4 aggregate-write-readPreference.json 0 0 4
aggregate.json 5 0 2 aggregate.json 5 0 2
bulkWrite-arrayFilters.json 0 3 0 bulkWrite-arrayFilters.json 3 0 0
bulkWrite-collation.json 0 2 0 bulkWrite-collation.json 0 2 0
bulkWrite-comment.json 2 0 1 bulkWrite-comment.json 2 0 1
bulkWrite-delete-hint-serverError.json 0 0 2 bulkWrite-delete-hint-serverError.json 0 0 2
@@ -141,7 +141,7 @@ findOneAndReplace-let.json 0 1 1
findOneAndReplace-rawdata.json 1 0 1 findOneAndReplace-rawdata.json 1 0 1
findOneAndReplace-upsert.json 2 2 0 findOneAndReplace-upsert.json 2 2 0
findOneAndReplace.json 4 2 0 findOneAndReplace.json 4 2 0
findOneAndUpdate-arrayFilters.json 0 3 0 findOneAndUpdate-arrayFilters.json 3 0 0
findOneAndUpdate-collation.json 0 1 0 findOneAndUpdate-collation.json 0 1 0
findOneAndUpdate-comment.json 0 2 1 findOneAndUpdate-comment.json 0 2 1
findOneAndUpdate-dots_and_dollars.json 0 0 4 findOneAndUpdate-dots_and_dollars.json 0 0 4
@@ -172,7 +172,7 @@ replaceOne-rawdata.json 1 0 1
replaceOne-sort.json 1 0 1 replaceOne-sort.json 1 0 1
replaceOne-validation.json 1 0 0 replaceOne-validation.json 1 0 0
replaceOne.json 5 0 0 replaceOne.json 5 0 0
updateMany-arrayFilters.json 0 3 0 updateMany-arrayFilters.json 3 0 0
updateMany-collation.json 0 1 0 updateMany-collation.json 0 1 0
updateMany-comment.json 2 0 1 updateMany-comment.json 2 0 1
updateMany-dots_and_dollars.json 0 0 4 updateMany-dots_and_dollars.json 0 0 4
@@ -183,7 +183,7 @@ updateMany-pipeline.json 0 1 0
updateMany-rawdata.json 1 0 1 updateMany-rawdata.json 1 0 1
updateMany-validation.json 1 0 0 updateMany-validation.json 1 0 0
updateMany.json 4 0 0 updateMany.json 4 0 0
updateOne-arrayFilters.json 0 5 0 updateOne-arrayFilters.json 5 0 0
updateOne-collation.json 0 1 0 updateOne-collation.json 0 1 0
updateOne-comment.json 2 0 1 updateOne-comment.json 2 0 1
updateOne-dots_and_dollars.json 0 0 4 updateOne-dots_and_dollars.json 0 0 4
@@ -209,9 +209,6 @@ aggregate-rawdata.json SKIP Aggregate with rawData option needs server >= 8.2.0
aggregate-write-readPreference.json SKIP * needs topology replicaset/sharded/load-balanced aggregate-write-readPreference.json SKIP * needs topology replicaset/sharded/load-balanced
aggregate.json SKIP aggregate with a document comment - pre 4.4 needs server <= 4.2.99 aggregate.json SKIP aggregate with a document comment - pre 4.4 needs server <= 4.2.99
aggregate.json SKIP aggregate with comment does not set comment on getMore - pre 4.4 needs server <= 4.3.99 aggregate.json SKIP aggregate with comment does not set comment on getMore - pre 4.4 needs server <= 4.3.99
bulkWrite-arrayFilters.json FAIL BulkWrite updateOne with arrayFilters MongoBulkWriteError: the positional operator '$[i]' in path 'y.$[i].b' is not implemented by this server
bulkWrite-arrayFilters.json FAIL BulkWrite updateMany with arrayFilters MongoBulkWriteError: the positional operator '$[i]' in path 'y.$[i].b' is not implemented by this server
bulkWrite-arrayFilters.json FAIL BulkWrite with arrayFilters MongoBulkWriteError: the positional operator '$[i]' in path 'y.$[i].b' is not implemented by this server
bulkWrite-collation.json FAIL BulkWrite with delete operations and collation bulkWrite.deletedCount: expected 4, got 0 bulkWrite-collation.json FAIL BulkWrite with delete operations and collation bulkWrite.deletedCount: expected 4, got 0
bulkWrite-collation.json FAIL BulkWrite with update operations and collation bulkWrite.matchedCount: expected 6, got 2 bulkWrite-collation.json FAIL BulkWrite with update operations and collation bulkWrite.matchedCount: expected 6, got 2
bulkWrite-comment.json SKIP BulkWrite with comment - pre 4.4 needs server <= 4.2.99 bulkWrite-comment.json SKIP BulkWrite with comment - pre 4.4 needs server <= 4.2.99
@@ -361,9 +358,6 @@ findOneAndReplace-upsert.json FAIL FindOneAndReplace when no documents match wit
findOneAndReplace-upsert.json FAIL FindOneAndReplace when no documents match with id specified with upsert returning the document after modification findOneAndReplace: expected a document, got null findOneAndReplace-upsert.json FAIL FindOneAndReplace when no documents match with id specified with upsert returning the document after modification findOneAndReplace: expected a document, got null
findOneAndReplace.json FAIL FindOneAndReplace when many documents match returning the document after modification findOneAndReplace.x: expected 32, got 22 findOneAndReplace.json FAIL FindOneAndReplace when many documents match returning the document after modification findOneAndReplace.x: expected 32, got 22
findOneAndReplace.json FAIL FindOneAndReplace when one document matches returning the document after modification findOneAndReplace.x: expected 32, got 22 findOneAndReplace.json FAIL FindOneAndReplace when one document matches returning the document after modification findOneAndReplace.x: expected 32, got 22
findOneAndUpdate-arrayFilters.json FAIL FindOneAndUpdate when no document matches arrayFilters MongoServerError: the positional operator '$[i]' in path 'y.$[i].b' is not implemented by this server
findOneAndUpdate-arrayFilters.json FAIL FindOneAndUpdate when one document matches arrayFilters MongoServerError: the positional operator '$[i]' in path 'y.$[i].b' is not implemented by this server
findOneAndUpdate-arrayFilters.json FAIL FindOneAndUpdate when multiple documents match arrayFilters MongoServerError: the positional operator '$[i]' in path 'y.$[i].b' is not implemented by this server
findOneAndUpdate-collation.json FAIL FindOneAndUpdate when many documents match with collation returning the document before modification findOneAndUpdate: expected a document, got null findOneAndUpdate-collation.json FAIL FindOneAndUpdate when many documents match with collation returning the document before modification findOneAndUpdate: expected a document, got null
findOneAndUpdate-comment.json FAIL findOneAndUpdate with string comment MongoServerError: update must be a document findOneAndUpdate-comment.json FAIL findOneAndUpdate with string comment MongoServerError: update must be a document
findOneAndUpdate-comment.json FAIL findOneAndUpdate with document comment MongoServerError: update must be a document findOneAndUpdate-comment.json FAIL findOneAndUpdate with document comment MongoServerError: update must be a document
@@ -403,9 +397,6 @@ replaceOne-let.json SKIP ReplaceOne with let option needs server >= 5.0
replaceOne-let.json FAIL ReplaceOne with let option unsupported (server-side error) replaceOne: expected an error, the operation succeeded replaceOne-let.json FAIL ReplaceOne with let option unsupported (server-side error) replaceOne: expected an error, the operation succeeded
replaceOne-rawdata.json SKIP ReplaceOne with rawData option needs server >= 8.2.0 replaceOne-rawdata.json SKIP ReplaceOne with rawData option needs server >= 8.2.0
replaceOne-sort.json SKIP ReplaceOne with sort option needs server >= 8.0 replaceOne-sort.json SKIP ReplaceOne with sort option needs server >= 8.0
updateMany-arrayFilters.json FAIL UpdateMany when no documents match arrayFilters MongoServerError: the positional operator '$[i]' in path 'y.$[i].b' is not implemented by this server
updateMany-arrayFilters.json FAIL UpdateMany when one document matches arrayFilters MongoServerError: the positional operator '$[i]' in path 'y.$[i].b' is not implemented by this server
updateMany-arrayFilters.json FAIL UpdateMany when multiple documents match arrayFilters MongoServerError: the positional operator '$[i]' in path 'y.$[i].b' is not implemented by this server
updateMany-collation.json FAIL UpdateMany when many documents match with collation updateMany.matchedCount: expected 2, got 1 updateMany-collation.json FAIL UpdateMany when many documents match with collation updateMany.matchedCount: expected 2, got 1
updateMany-comment.json SKIP UpdateMany with comment - pre 4.4 needs server <= 4.2.99 updateMany-comment.json SKIP UpdateMany with comment - pre 4.4 needs server <= 4.2.99
updateMany-dots_and_dollars.json SKIP Updating document to set top-level dollar-prefixed key on 5.0+ server needs server >= 5.0 updateMany-dots_and_dollars.json SKIP Updating document to set top-level dollar-prefixed key on 5.0+ server needs server >= 5.0
@@ -416,11 +407,6 @@ updateMany-let.json SKIP updateMany with let option needs server >= 5.0
updateMany-let.json FAIL updateMany with let option unsupported (server-side error) updateMany: error message "update spec requires u" does not contain "'update.let' is an unknown field" updateMany-let.json FAIL updateMany with let option unsupported (server-side error) updateMany: error message "update spec requires u" does not contain "'update.let' is an unknown field"
updateMany-pipeline.json FAIL UpdateMany using pipelines MongoServerError: update spec requires u updateMany-pipeline.json FAIL UpdateMany using pipelines MongoServerError: update spec requires u
updateMany-rawdata.json SKIP updateMany with rawData option needs server >= 8.2.0 updateMany-rawdata.json SKIP updateMany with rawData option needs server >= 8.2.0
updateOne-arrayFilters.json FAIL UpdateOne when no document matches arrayFilters MongoServerError: the positional operator '$[i]' in path 'y.$[i].b' is not implemented by this server
updateOne-arrayFilters.json FAIL UpdateOne when one document matches arrayFilters MongoServerError: the positional operator '$[i]' in path 'y.$[i].b' is not implemented by this server
updateOne-arrayFilters.json FAIL UpdateOne when multiple documents match arrayFilters MongoServerError: the positional operator '$[i]' in path 'y.$[i].b' is not implemented by this server
updateOne-arrayFilters.json FAIL UpdateOne when no documents match multiple arrayFilters MongoServerError: the positional operator '$[i]' in path 'y.$[i].c.$[j].d' is not implemented by this server
updateOne-arrayFilters.json FAIL UpdateOne when one document matches multiple arrayFilters MongoServerError: the positional operator '$[i]' in path 'y.$[i].c.$[j].d' is not implemented by this server
updateOne-collation.json FAIL UpdateOne when one document matches with collation updateOne.matchedCount: expected 1, got 0 updateOne-collation.json FAIL UpdateOne when one document matches with collation updateOne.matchedCount: expected 1, got 0
updateOne-comment.json SKIP UpdateOne with comment - pre 4.4 needs server <= 4.2.99 updateOne-comment.json SKIP UpdateOne with comment - pre 4.4 needs server <= 4.2.99
updateOne-dots_and_dollars.json SKIP Updating document to set top-level dollar-prefixed key on 5.0+ server needs server >= 5.0 updateOne-dots_and_dollars.json SKIP Updating document to set top-level dollar-prefixed key on 5.0+ server needs server >= 5.0