Aleksey Shakhmatov
f2844e7894
Every reply came back in a single batch with `cursor.id = 0`, `getMore` was a
stub answering an empty `nextBatch` on the literal namespace `test.$cmd`, and
nothing read `batchSize`. That caps the useful collection size at what fits in
one 48 MiB message, which is the opposite of the tens-of-GB target and the
reason M0 made whole-index scans stream: the streaming candidate generator
existed with no consumer that could suspend.
## What a cursor is allowed to remember
A cursor holds no lock between requests, so everything it saves has to survive
arbitrary concurrent mutation. Nothing here is a pointer, and the two things
that look like stable addresses are not: `reset_tree` re-creates node ids 0 and
1 as different nodes, and `rebuild_collection` moves every document. Three
sources, chosen by query shape, each with a different memory contract:
- **stream** -- an index-ordered walk resumed from a `(key, off)` anchor plus a
`(leaf, slot)` hint. O(key) state, so this is what lets a cursor walk a
collection larger than memory. Survives a rebuild, because a repack changes
no key.
- **offsets** -- the matched slab offsets a narrowed plan already materialized,
8 bytes each. Killed by a rebuild with `QueryPlanKilled`, because those
offsets now name unrelated bytes.
- **buffered** -- canonical BSON copies, for a sort no index provides and for
aggregate/listing output. Depends on nothing, which is what lets a listing
hold a cursor over a `$cmd.*` namespace no collection backs.
`Collection.layout_epoch` and `Index.epoch` are the invalidation tokens, both
checked as error returns rather than assertions since a client reaches them by
keeping a cursor open across maintenance.
## Resume
`resume_forward`/`resume_reverse` are O(1) while the hint holds and fall back to
an exact-order band walk bounded by `resume_walk_max`. Without the hint, `seek`
lands at the *start* of an equal-key band, so `sort({status: 1})` over three
distinct values across 10M documents would cost ~5e10 comparisons to drain.
Two hazards found by draining a collection while writing to it, neither
predictable from reading the code:
- A deleted anchor must resume at its *band position*, or the rest of an
equal-key band is silently dropped -- most of the collection on a
low-cardinality index. Hence `band_index`.
- On a **unique** index a same-key entry can only be the anchor rewritten, so
resuming at it returned updated documents twice. Observed as duplicate `_id`s
while updating underneath a drain.
## Protocol
Measured against mongod 8.3.7 rather than recalled, which corrected three
assumptions: a bare `getMore` does *not* inherit the find's `batchSize` (4998 of
5000 documents come back), a namespace mismatch is `Unauthorized` (13) not
`CursorNotFound`, and `CursorInUse` is 143 not 12051.
`internalQueryFindCommandBatchSize` is 101, `cursorTimeoutMillis` 600000,
`clientCursorMonitorFrequencySecs` 4.
The rule everything follows is **never look ahead**: a batch that met its target
leaves the cursor open even when the source is in fact exhausted, so four
documents at `batchSize: 2` take three commands. `limit` acts as an EOF source,
which is what makes `batchSize == limit` close in one round trip. `skip` is
consumed once. `batchSize: 0` returns an empty batch with a live cursor.
Cursor ids are `(nonce << 20) | slot`, always positive. The nonce is not
decoration: without it a recycled slot serves one client another's documents.
Cursors are not connection-pinned, since the driver spec allows a `getMore` on
any connection to the same server; they end at exhaustion, `killCursors`, or the
idle sweep (a second monitor fiber, separate from the TTL one because the
cadences differ by an order of magnitude and a TTL failure must not stop
reclamation). The registry is fixed-capacity and evicts the least recently used
cursor, whose client sees the same 43 an idle timeout gives.
Fixed alongside, because cursors are what expose them:
- `listCollections` reported `"<db>."` with an *empty* collection part, which
makes the driver throw client-side -- so it would have broken the moment its
cursor stopped being id 0. Now `<db>.$cmd.listCollections`, as mongod uses.
- `count` ignored `skip` and `limit` entirely.
- `wire.end_message` now bounds a reply by the 48 MiB we advertise rather than
by `maxInt(u32)`; a reply past what we told the client to expect is not a large
reply, it is a desynchronized connection.
- Two `codeName` strings were wrong: 72 is `InvalidOptions` (MongoDB has no
`InvalidArgument`), and 40324 reports as `Location40324`.
## Verification
Unit 160/160 in ReleaseFast and ReleaseSafe; `tests/e2e/e2e7.js` adds 86 cursor
checks across five phases (batching/lifecycle/errors, streaming across churn,
aggregate+listings+count, expiry+capacity, restart) and is self-contained
because cursor behaviour is only observable with non-default flags. No
regressions: e2e 49, e2e3 16, e2e4 17, e2e2 2, e2e6 72. Spec 168 pass / 124
fail, +5 against the previous scorecard.
Mutation-checked, per the repo's second ground rule: `hint_slot + 1`, the
`band_index` off-by-one, both epoch bumps, the id nonce, the at-least-one-
document rule, and `stream_shape` returning null each turn the intended test
red. One claim was withdrawn rather than kept -- swapping `std.mem.order` for
`cmp_prefix` in the band walk changes nothing observable, so the comment now
says so instead of asserting a check that does not hold.
129 lines
5.6 KiB
Markdown
129 lines
5.6 KiB
Markdown
# End-to-end tests with the official MongoDB Node.js driver
|
|
|
|
These exercise MultiforaDB from a real driver over TCP: full CRUD, query
|
|
operators, aggregation, error codes, concurrent clients, crash recovery, and
|
|
the whole lifecycle including server restarts.
|
|
|
|
## Setup
|
|
|
|
```sh
|
|
cd tests/e2e
|
|
npm init -y >/dev/null
|
|
npm install mongodb
|
|
```
|
|
|
|
## Run
|
|
|
|
Most suites expect a server running on port 27020:
|
|
|
|
```sh
|
|
zig build
|
|
zig-out/bin/multiforadb --port 27020 --db /tmp/mfdb-e2e.log --ttl-sweep-secs 1 &
|
|
|
|
node tests/e2e/e2e.js # CRUD + operators + aggregate + errors (29 checks)
|
|
node tests/e2e/e2e2.js concurrent # 8 clients: 4 writers + 4 readers (2 checks)
|
|
node tests/e2e/e2e2.js crash-a # write 50 docs, then kill -9 the server
|
|
node tests/e2e/e2e2.js crash-b # restart and verify all 50 survived
|
|
node tests/e2e/e2e3.js # secondary indexes: unique/sparse/compound (16 checks)
|
|
node tests/e2e/e2e4.js # TTL indexes: expiry + rejected specs (15 checks)
|
|
```
|
|
|
|
`e2e4.js` needs the server started with `--ttl-sweep-secs 1` (the default is
|
|
60 seconds); the other suites do not care about the flag.
|
|
|
|
`e2e6.js` is the full-lifecycle suite and is self-contained: it spawns its
|
|
own server on port 27220 with a fresh log, runs the whole feature surface,
|
|
restarts the server twice (graceful SIGTERM, then kill -9 mid-write) and
|
|
verifies everything survived:
|
|
|
|
```sh
|
|
node tests/e2e/e2e6.js # 73 checks, ~15 s, needs no running server
|
|
E2E6_PORT=27300 node tests/e2e/e2e6.js # different port if 27220 is taken
|
|
```
|
|
|
|
`e2e7.js` is the cursor suite, self-contained for a different reason: cursor
|
|
behaviour is only observable with non-default flags. It spawns three servers in
|
|
turn -- default flags for batching/streaming/aggregate, then
|
|
`--cursor-timeout-ms 800 --cursor-sweep-secs 1 --max-open-cursors 4` for idle
|
|
expiry and registry capacity, then a restart on the same database to confirm a
|
|
cursor does not survive one. Most of it uses raw `runCommand`, because the
|
|
driver hides `cursor.id` and that is the thing under test:
|
|
|
|
```sh
|
|
node tests/e2e/e2e7.js # 86 checks, needs no running server
|
|
E2E7_PORT=27310 node tests/e2e/e2e7.js # different port if 27230 is taken
|
|
```
|
|
|
|
Rebuild with `zig build` after any change under `src/` before restarting the
|
|
server: `zig build test` compiles the test binary only and leaves
|
|
`zig-out/bin/multiforadb` stale, so the suites keep running against the old
|
|
rules and report failures that the source no longer explains.
|
|
|
|
`e2e2.js concurrent` is safe to repeat against a running server (it drops its
|
|
collection first); `crash-a`/`crash-b` are two halves of one scenario.
|
|
|
|
## Multi-GB collections: `big.js`
|
|
|
|
`big.js` is a load harness, not a pass/fail suite: it spawns a server, bulk
|
|
loads up to ~5 GB, and reports insert throughput, the compaction behavior,
|
|
server RSS, per-operation latencies, reopen (replay) time, and kill -9
|
|
durability.
|
|
|
|
```sh
|
|
node tests/e2e/big.js --quick # 268 MB smoke run
|
|
node tests/e2e/big.js --size 5g --doc-size 128k --oid --batch 200 \
|
|
--compact-threshold 2g # ~5 GB, 40k docs
|
|
```
|
|
|
|
Options: `--size`/`--doc-size`/`--batch` (k/m/g suffixes), `--oid`
|
|
(ObjectId `_id`s — see below), `--index <field>` (secondary index before
|
|
loading), `--compact-threshold <bytes>` (passed to the server),
|
|
`--port`, `--keep` (keep the db file).
|
|
|
|
Measured behavior (all documented in the top-level README):
|
|
|
|
- **Build in ReleaseFast** — `zig build` defaults to it; a Debug server is
|
|
10-200x slower on every path.
|
|
- **Insert throughput collapses under the default 16 MiB compaction
|
|
threshold**: every ~16 MB of writes rewrites the whole log with one fsync
|
|
per record (O(n²) total). With `--compact-threshold 2g` the rate stays
|
|
flat (hundreds of MB/s at 128 KB docs in ReleaseFast). Raise the
|
|
threshold for bulk loads.
|
|
- **`findOne({_id})` is O(1) only for ObjectId `_id`s.** Integer `_id`s are
|
|
serialization-ambiguous (int32/int64/double compare equal but hash
|
|
differently), so the docs-map fast path is skipped and every lookup is a
|
|
full scan. Use the driver's default ObjectId ids on big collections.
|
|
- **The engine holds everything in RAM**: ~1-1.2x the data size at 128 KB
|
|
docs (more at 16 KB docs, where per-document arena overhead dominates).
|
|
A 5 GB collection needs roughly 6-7 GB of RAM.
|
|
- Reopen of a 5 GB log replays in ~10 s (ReleaseFast); every committed
|
|
write survives kill -9.
|
|
|
|
## Comparing against real MongoDB: `compare.js` + `compare-run.sh`
|
|
|
|
```sh
|
|
bash tests/e2e/compare-run.sh [size] [doc-size] # e.g. 1g 16k
|
|
```
|
|
|
|
Starts mongod (`brew install mongodb-community`) on :27018 and MultiforaDB
|
|
on :27019, runs the same driver workload against each (durable writes:
|
|
MultiforaDB fsyncs per command, mongod runs with `j: true`), measures kill -9
|
|
reopen for both, and prints a side-by-side table. `compare.js` alone runs
|
|
one side (see its `--help`-style header comment).
|
|
|
|
### Iteration-to-iteration comparison: `bench-run.sh` + `concurrent.js`
|
|
|
|
```sh
|
|
bash tests/e2e/bench-run.sh [size] [doc-size] ["clients..."] # e.g. 1g 16k "1 4 8 16 32"
|
|
```
|
|
|
|
Runs the main suite (`compare-run.sh`) plus a concurrent durable-write
|
|
comparison (`concurrent.js`, N clients each doing sequential `insertOne`
|
|
with `{w:1, j:true}` — the group-commit path under real contention), then
|
|
writes a machine-readable, versioned report to
|
|
`tests/e2e/results/bench-<timestamp>.txt` and prints a diff of the
|
|
MultiforaDB numbers against the previous run (`results/bench-latest.txt`).
|
|
The report has `[main]` / `[concurrency]` / `[meta]` sections with
|
|
`name<TAB>value` rows; `bench-run.sh 1g 16k` reproduces the phase8 gate
|
|
(see `results/phase8.txt`).
|