cursors: server-side cursors for find, aggregate and the listing commands
Every reply came back in a single batch with `cursor.id = 0`, `getMore` was a
stub answering an empty `nextBatch` on the literal namespace `test.$cmd`, and
nothing read `batchSize`. That caps the useful collection size at what fits in
one 48 MiB message, which is the opposite of the tens-of-GB target and the
reason M0 made whole-index scans stream: the streaming candidate generator
existed with no consumer that could suspend.
## What a cursor is allowed to remember
A cursor holds no lock between requests, so everything it saves has to survive
arbitrary concurrent mutation. Nothing here is a pointer, and the two things
that look like stable addresses are not: `reset_tree` re-creates node ids 0 and
1 as different nodes, and `rebuild_collection` moves every document. Three
sources, chosen by query shape, each with a different memory contract:
- **stream** -- an index-ordered walk resumed from a `(key, off)` anchor plus a
`(leaf, slot)` hint. O(key) state, so this is what lets a cursor walk a
collection larger than memory. Survives a rebuild, because a repack changes
no key.
- **offsets** -- the matched slab offsets a narrowed plan already materialized,
8 bytes each. Killed by a rebuild with `QueryPlanKilled`, because those
offsets now name unrelated bytes.
- **buffered** -- canonical BSON copies, for a sort no index provides and for
aggregate/listing output. Depends on nothing, which is what lets a listing
hold a cursor over a `$cmd.*` namespace no collection backs.
`Collection.layout_epoch` and `Index.epoch` are the invalidation tokens, both
checked as error returns rather than assertions since a client reaches them by
keeping a cursor open across maintenance.
## Resume
`resume_forward`/`resume_reverse` are O(1) while the hint holds and fall back to
an exact-order band walk bounded by `resume_walk_max`. Without the hint, `seek`
lands at the *start* of an equal-key band, so `sort({status: 1})` over three
distinct values across 10M documents would cost ~5e10 comparisons to drain.
Two hazards found by draining a collection while writing to it, neither
predictable from reading the code:
- A deleted anchor must resume at its *band position*, or the rest of an
equal-key band is silently dropped -- most of the collection on a
low-cardinality index. Hence `band_index`.
- On a **unique** index a same-key entry can only be the anchor rewritten, so
resuming at it returned updated documents twice. Observed as duplicate `_id`s
while updating underneath a drain.
## Protocol
Measured against mongod 8.3.7 rather than recalled, which corrected three
assumptions: a bare `getMore` does *not* inherit the find's `batchSize` (4998 of
5000 documents come back), a namespace mismatch is `Unauthorized` (13) not
`CursorNotFound`, and `CursorInUse` is 143 not 12051.
`internalQueryFindCommandBatchSize` is 101, `cursorTimeoutMillis` 600000,
`clientCursorMonitorFrequencySecs` 4.
The rule everything follows is **never look ahead**: a batch that met its target
leaves the cursor open even when the source is in fact exhausted, so four
documents at `batchSize: 2` take three commands. `limit` acts as an EOF source,
which is what makes `batchSize == limit` close in one round trip. `skip` is
consumed once. `batchSize: 0` returns an empty batch with a live cursor.
Cursor ids are `(nonce << 20) | slot`, always positive. The nonce is not
decoration: without it a recycled slot serves one client another's documents.
Cursors are not connection-pinned, since the driver spec allows a `getMore` on
any connection to the same server; they end at exhaustion, `killCursors`, or the
idle sweep (a second monitor fiber, separate from the TTL one because the
cadences differ by an order of magnitude and a TTL failure must not stop
reclamation). The registry is fixed-capacity and evicts the least recently used
cursor, whose client sees the same 43 an idle timeout gives.
Fixed alongside, because cursors are what expose them:
- `listCollections` reported `"<db>."` with an *empty* collection part, which
makes the driver throw client-side -- so it would have broken the moment its
cursor stopped being id 0. Now `<db>.$cmd.listCollections`, as mongod uses.
- `count` ignored `skip` and `limit` entirely.
- `wire.end_message` now bounds a reply by the 48 MiB we advertise rather than
by `maxInt(u32)`; a reply past what we told the client to expect is not a large
reply, it is a desynchronized connection.
- Two `codeName` strings were wrong: 72 is `InvalidOptions` (MongoDB has no
`InvalidArgument`), and 40324 reports as `Location40324`.
## Verification
Unit 160/160 in ReleaseFast and ReleaseSafe; `tests/e2e/e2e7.js` adds 86 cursor
checks across five phases (batching/lifecycle/errors, streaming across churn,
aggregate+listings+count, expiry+capacity, restart) and is self-contained
because cursor behaviour is only observable with non-default flags. No
regressions: e2e 49, e2e3 16, e2e4 17, e2e2 2, e2e6 72. Spec 168 pass / 124
fail, +5 against the previous scorecard.
Mutation-checked, per the repo's second ground rule: `hint_slot + 1`, the
`band_index` off-by-one, both epoch bumps, the id nonce, the at-least-one-
document rule, and `stream_shape` returning null each turn the intended test
red. One claim was withdrawn rather than kept -- swapping `std.mem.order` for
`cmp_prefix` in the band walk changes nothing observable, so the comment now
says so instead of asserting a check that does not hold.
This commit is contained in:
@@ -41,6 +41,19 @@ node tests/e2e/e2e6.js # 73 checks, ~15 s, needs no running server
|
||||
E2E6_PORT=27300 node tests/e2e/e2e6.js # different port if 27220 is taken
|
||||
```
|
||||
|
||||
`e2e7.js` is the cursor suite, self-contained for a different reason: cursor
|
||||
behaviour is only observable with non-default flags. It spawns three servers in
|
||||
turn -- default flags for batching/streaming/aggregate, then
|
||||
`--cursor-timeout-ms 800 --cursor-sweep-secs 1 --max-open-cursors 4` for idle
|
||||
expiry and registry capacity, then a restart on the same database to confirm a
|
||||
cursor does not survive one. Most of it uses raw `runCommand`, because the
|
||||
driver hides `cursor.id` and that is the thing under test:
|
||||
|
||||
```sh
|
||||
node tests/e2e/e2e7.js # 86 checks, needs no running server
|
||||
E2E7_PORT=27310 node tests/e2e/e2e7.js # different port if 27230 is taken
|
||||
```
|
||||
|
||||
Rebuild with `zig build` after any change under `src/` before restarting the
|
||||
server: `zig build test` compiles the test binary only and leaves
|
||||
`zig-out/bin/multiforadb` stale, so the suites keep running against the old
|
||||
|
||||
474
tests/e2e/e2e7.js
Normal file
474
tests/e2e/e2e7.js
Normal file
@@ -0,0 +1,474 @@
|
||||
// E2E part 7: server-side cursors, self-contained.
|
||||
//
|
||||
// Spawns its own multiforadb servers, because cursor behaviour is only
|
||||
// observable with non-default flags (a short idle timeout, a tiny registry) and
|
||||
// with raw `runCommand` — the driver hides `cursor.id`, which is the thing under
|
||||
// test.
|
||||
//
|
||||
// node tests/e2e/e2e7.js
|
||||
//
|
||||
// Env: E2E7_PORT listen port (default 27230)
|
||||
// MFDB_BIN server binary (default ../../zig-out/bin/multiforadb)
|
||||
// E2E7_KEEP keep the log files after the run
|
||||
//
|
||||
// The one rule most of this file is about: **never look ahead.** A batch ends
|
||||
// either because it reached its target — cursor stays open — or because the
|
||||
// source reported EOF, and only then does the cursor close with `id: 0`. So four
|
||||
// documents at `batchSize: 2` require a third command answering an empty
|
||||
// `nextBatch` with `id: 0`. That empty terminal batch is correct, and it is what
|
||||
// real mongod does (measured, not assumed — see checks 5 and 6).
|
||||
const { MongoClient, Long } = require('mongodb');
|
||||
const { spawn } = require('child_process');
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
|
||||
const PORT = Number(process.env.E2E7_PORT || 27230);
|
||||
const BIN = process.env.MFDB_BIN || path.resolve(__dirname, '../../zig-out/bin/multiforadb');
|
||||
const DBFILE = path.resolve(__dirname, '../../.zig-cache/e2e7-cursors.log');
|
||||
const URL = `mongodb://127.0.0.1:${PORT}`;
|
||||
|
||||
const results = [];
|
||||
function check(name, cond, detail = '') {
|
||||
results.push({ name, ok: !!cond, detail: String(detail) });
|
||||
if (!cond) console.error(` x ${name} ${detail}`);
|
||||
}
|
||||
function eq(name, got, want) {
|
||||
const ok = JSON.stringify(got) === JSON.stringify(want);
|
||||
check(name, ok, ok ? '' : `got ${JSON.stringify(got)} want ${JSON.stringify(want)}`);
|
||||
}
|
||||
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
|
||||
|
||||
/// The error code a command produces, or 0 when it succeeds.
|
||||
async function codeOf(fn) {
|
||||
try {
|
||||
await fn();
|
||||
return 0;
|
||||
} catch (e) {
|
||||
return e.code === undefined ? -1 : e.code;
|
||||
}
|
||||
}
|
||||
|
||||
let server = null;
|
||||
let serverLog = '';
|
||||
let serverDead = false;
|
||||
|
||||
function cleanup() {
|
||||
if (server && !serverDead) {
|
||||
try { server.kill('SIGKILL'); } catch {}
|
||||
}
|
||||
}
|
||||
process.on('exit', cleanup);
|
||||
process.on('SIGINT', () => { cleanup(); process.exit(130); });
|
||||
process.on('SIGTERM', () => { cleanup(); process.exit(143); });
|
||||
|
||||
function startServer(args, fresh = true) {
|
||||
return new Promise((resolve, reject) => {
|
||||
if (fresh) {
|
||||
fs.rmSync(DBFILE, { force: true });
|
||||
fs.rmSync(DBFILE + '.data', { force: true });
|
||||
}
|
||||
serverDead = false;
|
||||
serverLog = '';
|
||||
server = spawn(BIN, ['--port', String(PORT), '--db', DBFILE, ...args], {
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
});
|
||||
server.stdout.on('data', (d) => (serverLog += d));
|
||||
server.stderr.on('data', (d) => (serverLog += d));
|
||||
server.on('error', (e) => reject(new Error(`cannot start ${BIN}: ${e.message}`)));
|
||||
server.on('exit', (code, sig) => {
|
||||
serverDead = true;
|
||||
if (code !== null && sig === null) serverLog += `\n[child exited rc=${code}]`;
|
||||
});
|
||||
const deadline = Date.now() + 15000;
|
||||
(async () => {
|
||||
while (Date.now() < deadline) {
|
||||
if (serverDead) {
|
||||
reject(new Error(`server exited during start (port ${PORT} busy?)\n${serverLog}`));
|
||||
return;
|
||||
}
|
||||
const c = new MongoClient(URL, { serverSelectionTimeoutMS: 1000 });
|
||||
try {
|
||||
await c.connect();
|
||||
await c.db('admin').command({ ping: 1 });
|
||||
await c.close();
|
||||
return resolve();
|
||||
} catch {
|
||||
try { await c.close(); } catch {}
|
||||
await sleep(100);
|
||||
}
|
||||
}
|
||||
reject(new Error(`server did not come up on :${PORT}\n${serverLog}`));
|
||||
})();
|
||||
});
|
||||
}
|
||||
|
||||
async function stopServer(sig = 'SIGTERM') {
|
||||
if (!server) return;
|
||||
const exited = new Promise((r) => server.once('exit', r));
|
||||
server.kill(sig);
|
||||
await Promise.race([exited, sleep(5000)]);
|
||||
serverDead = true;
|
||||
server = null;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Phase A — batching, lifecycle and errors, on default flags
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
async function phaseA(db) {
|
||||
const col = db.collection('c');
|
||||
await col.deleteMany({});
|
||||
await col.insertMany([...Array(250)].map((_, i) => ({ _id: i + 1, x: i })));
|
||||
|
||||
// 1. A cursor is a real cursor: nonzero id, a namespace with both parts.
|
||||
let r = await db.command({ find: 'c', filter: {}, batchSize: 2 });
|
||||
eq('1 batchSize 2 returns 2', r.cursor.firstBatch.length, 2);
|
||||
check('1 cursor id is nonzero', r.cursor.id > 0, r.cursor.id);
|
||||
eq('1 ns is db.coll', r.cursor.ns, 'e2e7.c');
|
||||
|
||||
// 2. The default first batch is 101, MongoDB's own
|
||||
// internalQueryFindCommandBatchSize.
|
||||
eq('2 default first batch is 101', (await db.command({ find: 'c', filter: {} })).cursor.firstBatch.length, 101);
|
||||
|
||||
// 3. A getMore naming no batchSize is bounded by bytes, not by the batchSize
|
||||
// the cursor was created with. Measured against mongod 8.3.7: find with
|
||||
// batchSize 2 then a bare getMore returns 4998 of 5000 documents.
|
||||
const id3 = (await db.command({ find: 'c', filter: {}, batchSize: 2 })).cursor.id;
|
||||
let g = await db.command({ getMore: id3, collection: 'c' });
|
||||
eq('3 a bare getMore drains the rest', g.cursor.nextBatch.length, 248);
|
||||
eq('3 and closes at EOF', String(g.cursor.id), '0');
|
||||
|
||||
// 4. A getMore's batchSize applies to that batch only.
|
||||
const id4 = (await db.command({ find: 'c', filter: {}, batchSize: 2 })).cursor.id;
|
||||
g = await db.command({ getMore: id4, collection: 'c', batchSize: 3 });
|
||||
eq('4 getMore batchSize 3', g.cursor.nextBatch.map((d) => d._id), [3, 4, 5]);
|
||||
check('4 still open', g.cursor.id > 0);
|
||||
|
||||
// 5. No look-ahead: 4 documents at batchSize 2 needs a third command whose
|
||||
// nextBatch is empty. Closing on "batch full and source dry" would break
|
||||
// the command counts the pinned spec suites assert.
|
||||
const four = db.collection('four');
|
||||
await four.deleteMany({});
|
||||
await four.insertMany([1, 2, 3, 4].map((i) => ({ _id: i })));
|
||||
r = await db.command({ find: 'four', filter: {}, batchSize: 2 });
|
||||
g = await db.command({ getMore: r.cursor.id, collection: 'four', batchSize: 2 });
|
||||
eq('5 second full batch is 2 documents', g.cursor.nextBatch.length, 2);
|
||||
check('5 and leaves the cursor open', g.cursor.id > 0);
|
||||
g = await db.command({ getMore: g.cursor.id, collection: 'four', batchSize: 2 });
|
||||
eq('5 terminal batch is empty and closed', [g.cursor.nextBatch.length, String(g.cursor.id)], [0, '0']);
|
||||
|
||||
// 6. limit is an EOF source, so the batch that exhausts it also closes the
|
||||
// cursor. This is why the driver sends batchSize = limit + 1.
|
||||
r = await db.command({ find: 'c', filter: {}, sort: { _id: 1 }, limit: 4, batchSize: 5 });
|
||||
eq('6 limit 4 batchSize 5 closes in one reply', String(r.cursor.id), '0');
|
||||
eq('6 and returns exactly the limit', r.cursor.firstBatch.map((d) => d._id), [1, 2, 3, 4]);
|
||||
r = await db.command({ find: 'c', filter: {}, sort: { _id: 1 }, limit: 4, batchSize: 2 });
|
||||
check('6 limit 4 batchSize 2 stays open', r.cursor.id > 0);
|
||||
g = await db.command({ getMore: r.cursor.id, collection: 'c', batchSize: 2 });
|
||||
eq('6 the batch reaching the limit closes', String(g.cursor.id), '0');
|
||||
eq('6 across batches, limit still honoured', g.cursor.nextBatch.map((d) => d._id), [3, 4]);
|
||||
|
||||
// 7. skip is consumed once, at creation, and never re-applied.
|
||||
r = await db.command({ find: 'c', filter: {}, skip: 20, batchSize: 3 });
|
||||
eq('7 skip 20 starts at 21', r.cursor.firstBatch.map((d) => d._id), [21, 22, 23]);
|
||||
g = await db.command({ getMore: r.cursor.id, collection: 'c', batchSize: 3 });
|
||||
eq('7 skip not re-applied on getMore', g.cursor.nextBatch.map((d) => d._id), [24, 25, 26]);
|
||||
|
||||
// 8. batchSize 0 is a real request for an empty batch with a live cursor, not
|
||||
// "unbounded". Drivers use it to obtain a cursor cheaply. Nothing may be
|
||||
// consumed.
|
||||
r = await db.command({ find: 'c', filter: {}, batchSize: 0 });
|
||||
eq('8 batchSize 0 returns nothing', r.cursor.firstBatch.length, 0);
|
||||
check('8 but a live cursor', r.cursor.id > 0);
|
||||
g = await db.command({ getMore: r.cursor.id, collection: 'c', batchSize: 1 });
|
||||
eq('8 nothing was consumed', g.cursor.nextBatch.map((d) => d._id), [1]);
|
||||
|
||||
// 9. singleBatch, and its wire-legacy form, a negative limit.
|
||||
eq('9 singleBatch closes', String((await db.command({ find: 'c', filter: {}, batchSize: 2, singleBatch: true })).cursor.id), '0');
|
||||
r = await db.command({ find: 'c', filter: {}, limit: -3 });
|
||||
eq('9 negative limit is one batch', [r.cursor.firstBatch.length, String(r.cursor.id)], [3, '0']);
|
||||
|
||||
// 10. A cursor is not pinned to the connection that created it: the driver
|
||||
// spec allows a getMore from any connection to the same server.
|
||||
const other = new MongoClient(URL);
|
||||
await other.connect();
|
||||
const shared = (await db.command({ find: 'c', filter: {}, batchSize: 2 })).cursor.id;
|
||||
eq('10 getMore from another connection', await codeOf(() => other.db('e2e7').command({ getMore: shared, collection: 'c', batchSize: 2 })), 0);
|
||||
await other.close();
|
||||
|
||||
// 11. A getMore naming the wrong collection is Unauthorized (13), not 43, and
|
||||
// leaves the cursor alive — the request is wrong, not the cursor.
|
||||
// Measured against mongod, which answers exactly this code.
|
||||
const live = (await db.command({ find: 'c', filter: {}, batchSize: 2 })).cursor.id;
|
||||
eq('11 wrong collection is Unauthorized 13', await codeOf(() => db.command({ getMore: live, collection: 'four' })), 13);
|
||||
eq('11 the cursor survived it', (await db.command({ getMore: live, collection: 'c', batchSize: 1 })).cursor.nextBatch.length, 1);
|
||||
|
||||
// 12. killCursors: all four arrays, and the right partitioning.
|
||||
let k = await db.command({ killCursors: 'c', cursors: [live] });
|
||||
eq('12 a live cursor is killed', k.cursorsKilled.map(String), [String(live)]);
|
||||
check('12 all four arrays present', ['cursorsKilled', 'cursorsNotFound', 'cursorsAlive', 'cursorsUnknown'].every((f) => Array.isArray(k[f])), Object.keys(k).join(','));
|
||||
k = await db.command({ killCursors: 'c', cursors: [live] });
|
||||
eq('12 killing it twice reports notFound', k.cursorsNotFound.map(String), [String(live)]);
|
||||
const other_ns = (await db.command({ find: 'four', filter: {}, batchSize: 1 })).cursor.id;
|
||||
k = await db.command({ killCursors: 'c', cursors: [other_ns] });
|
||||
eq('12 a wrong-namespace id reports notFound', k.cursorsNotFound.map(String), [String(other_ns)]);
|
||||
eq('12 and that cursor still lives', await codeOf(() => db.command({ getMore: other_ns, collection: 'four', batchSize: 1 })), 0);
|
||||
|
||||
// 13. Malformed and unknown ids.
|
||||
eq('13 getMore after kill is 43', await codeOf(() => db.command({ getMore: live, collection: 'c' })), 43);
|
||||
eq('13 id 0 is 43', await codeOf(() => db.command({ getMore: Long.fromNumber(0), collection: 'c' })), 43);
|
||||
eq('13 an unknown id is 43', await codeOf(() => db.command({ getMore: Long.fromString('987654321'), collection: 'c' })), 43);
|
||||
eq('13 a non-numeric id is TypeMismatch 14', await codeOf(() => db.command({ getMore: 'nope', collection: 'c' })), 14);
|
||||
eq('13 a missing collection is BadValue 2', await codeOf(() => db.command({ getMore: Long.fromNumber(1) })), 2);
|
||||
|
||||
// 14. tailable is refused, which is parity: mongod rejects it on a non-capped
|
||||
// collection and this engine has none. Ignoring it would make a driver's
|
||||
// tail loop exit, which the application reads as data loss.
|
||||
eq('14 tailable is BadValue 2', await codeOf(() => db.command({ find: 'c', filter: {}, tailable: true })), 2);
|
||||
eq('14 awaitData alone is BadValue 2', await codeOf(() => db.command({ find: 'c', filter: {}, awaitData: true })), 2);
|
||||
|
||||
// 15. The driver's own iteration, which is the point of all of the above.
|
||||
const ids = (await col.find({}).batchSize(7).toArray()).map((d) => d._id);
|
||||
eq('15 driver drains 250 at batchSize 7', ids.length, 250);
|
||||
eq('15 no duplicates and no gaps', [new Set(ids).size, Math.min(...ids), Math.max(...ids)], [250, 1, 250]);
|
||||
eq('15 sort+skip+limit unchanged', (await col.find({ _id: { $gt: 2 } }, { sort: { _id: 1 }, skip: 2, limit: 2 }).toArray()).map((d) => d._id), [5, 6]);
|
||||
// A sort no index provides must materialize; it still has to drain correctly.
|
||||
eq('15 an unindexed sort drains in order', (await col.find({}, { sort: { x: -1 } }).batchSize(10).toArray()).map((d) => d.x)[0], 249);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Phase B — streaming cursors: resume across writes, and what survives a rebuild
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
async function phaseB(db) {
|
||||
const col = db.collection('s');
|
||||
await col.deleteMany({});
|
||||
await col.insertMany([...Array(300)].map((_, i) => ({ _id: i + 1, a: i % 5, pad: 'q'.repeat(200) })));
|
||||
|
||||
// 16. A whole-index walk holds a key, not a list, so it resumes across writes
|
||||
// that move documents. The bug this caught: an update rewrites a document
|
||||
// to a new offset, and resuming by band position returned it twice.
|
||||
let r = await db.command({ find: 's', filter: {}, batchSize: 10 });
|
||||
const seen = new Set(r.cursor.firstBatch.map((d) => d._id));
|
||||
let dupes = 0;
|
||||
let id = r.cursor.id;
|
||||
let rounds = 0;
|
||||
let errored = 0;
|
||||
while (String(id) !== '0' && rounds++ < 200) {
|
||||
// Churn between every batch: updates rewrite documents, which both moves
|
||||
// them in the slab and can split leaves.
|
||||
await col.updateMany({ _id: { $lt: 60 } }, { $inc: { n: 1 } });
|
||||
let g;
|
||||
try {
|
||||
g = await db.command({ getMore: id, collection: 's', batchSize: 10 });
|
||||
} catch (e) {
|
||||
errored = e.code;
|
||||
break;
|
||||
}
|
||||
for (const d of g.cursor.nextBatch) {
|
||||
if (seen.has(d._id)) dupes++;
|
||||
seen.add(d._id);
|
||||
}
|
||||
id = g.cursor.id;
|
||||
}
|
||||
eq('16 draining across churn did not error', errored, 0);
|
||||
eq('16 no document came back twice', dupes, 0);
|
||||
eq('16 every document was returned', seen.size, 300);
|
||||
check('16 and nothing outside the collection', [...seen].every((v) => v >= 1 && v <= 300));
|
||||
|
||||
// 17. Both directions stream, over the _id_ index and a secondary one.
|
||||
eq('17 ascending _id sort drains', (await col.find({}, { sort: { _id: 1 } }).batchSize(9).toArray()).length, 300);
|
||||
const desc = (await col.find({}, { sort: { _id: -1 } }).batchSize(9).toArray()).map((d) => d._id);
|
||||
eq('17 descending drains in order', [desc.length, desc[0], desc[299]], [300, 300, 1]);
|
||||
await col.createIndex({ a: 1 });
|
||||
const bya = await col.find({}, { sort: { a: 1 } }).batchSize(11).toArray();
|
||||
eq('17 a secondary-index sort drains', bya.length, 300);
|
||||
check('17 and in the index order', bya.every((d, i) => i === 0 || bya[i - 1].a <= d.a));
|
||||
|
||||
// 18. Dropping the index a stream is following cannot be resumed — the walk
|
||||
// has nothing left to walk. That must be a clean error, not garbage.
|
||||
await col.createIndex({ b: 1 });
|
||||
const onb = (await db.command({ find: 's', filter: {}, sort: { b: 1 }, batchSize: 3 })).cursor.id;
|
||||
await col.dropIndex('b_1');
|
||||
eq('18 dropping the streamed index is 175', await codeOf(() => db.command({ getMore: onb, collection: 's', batchSize: 3 })), 175);
|
||||
|
||||
// 19. Dropping the collection kills every kind of cursor.
|
||||
const doomed = (await db.command({ find: 's', filter: {}, batchSize: 3 })).cursor.id;
|
||||
await col.drop();
|
||||
const dc = await codeOf(() => db.command({ getMore: doomed, collection: 's', batchSize: 3 }));
|
||||
check('19 dropping the collection kills the cursor', dc === 175 || dc === 43, dc);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Phase C — aggregate, the listing commands, and count
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
async function phaseC(db) {
|
||||
const col = db.collection('g');
|
||||
await col.deleteMany({});
|
||||
await col.insertMany([...Array(250)].map((_, i) => ({ _id: i + 1, g: i % 40, v: i })));
|
||||
|
||||
// 20. aggregate batches through cursor.batchSize; a bare cursor is the default.
|
||||
let r = await db.command({ aggregate: 'g', pipeline: [], cursor: { batchSize: 3 } });
|
||||
eq('20 aggregate batchSize 3', r.cursor.firstBatch.length, 3);
|
||||
check('20 aggregate cursor is real', r.cursor.id > 0);
|
||||
eq('20 aggregate ns', r.cursor.ns, 'e2e7.g');
|
||||
const g20 = await db.command({ getMore: r.cursor.id, collection: 'g', batchSize: 5 });
|
||||
eq('20 aggregate getMore continues', g20.cursor.nextBatch.map((d) => d._id), [4, 5, 6, 7, 8]);
|
||||
eq('20 bare cursor defaults to 101', (await db.command({ aggregate: 'g', pipeline: [], cursor: {} })).cursor.firstBatch.length, 101);
|
||||
eq('20 driver aggregate drains', (await col.aggregate([], { batchSize: 7 }).toArray()).length, 250);
|
||||
const groups = await col.aggregate([{ $group: { _id: '$g', n: { $sum: 1 } } }], { batchSize: 6 }).toArray();
|
||||
eq('20 $group drains across batches', [groups.length, groups.reduce((a, x) => a + x.n, 0)], [40, 250]);
|
||||
eq('20 $count stage', await col.aggregate([{ $count: 'total' }]).toArray(), [{ total: 250 }]);
|
||||
|
||||
// 21. listCollections' namespace. It used to be "<db>." with an empty
|
||||
// collection part, and the driver throws client-side on a namespace like
|
||||
// that — so the moment the cursor stopped being id 0 it would have broken.
|
||||
for (let i = 0; i < 12; i++) await db.createCollection('k' + i);
|
||||
r = await db.command({ listCollections: 1, cursor: { batchSize: 4 } });
|
||||
eq('21 listCollections ns has a collection part', r.cursor.ns, 'e2e7.$cmd.listCollections');
|
||||
eq('21 listCollections honours batchSize', r.cursor.firstBatch.length, 4);
|
||||
check('21 listCollections cursor is real', r.cursor.id > 0);
|
||||
const g21 = await db.command({ getMore: r.cursor.id, collection: '$cmd.listCollections', batchSize: 100 });
|
||||
check('21 its getMore works', g21.cursor.nextBatch.length >= 8, g21.cursor.nextBatch.length);
|
||||
const listed = await db.listCollections({}, { batchSize: 3 }).toArray();
|
||||
check('21 driver listCollections drains', listed.length >= 13, listed.length);
|
||||
|
||||
// 22. listIndexes.
|
||||
await col.createIndex({ v: 1 });
|
||||
await col.createIndex({ g: 1 });
|
||||
await col.createIndex({ v: -1, g: 1 });
|
||||
r = await db.command({ listIndexes: 'g', cursor: { batchSize: 2 } });
|
||||
eq('22 listIndexes honours batchSize', r.cursor.firstBatch.length, 2);
|
||||
eq('22 listIndexes ns', r.cursor.ns, 'e2e7.g');
|
||||
eq('22 driver listIndexes drains', (await col.listIndexes({ batchSize: 1 }).toArray()).length, 4);
|
||||
|
||||
// 23. count honoured neither skip nor limit before, which made
|
||||
// countDocuments(f, {limit}) a silent wrong answer.
|
||||
eq('23 count plain', (await db.command({ count: 'g' })).n, 250);
|
||||
eq('23 count limit', (await db.command({ count: 'g', limit: 10 })).n, 10);
|
||||
eq('23 count skip', (await db.command({ count: 'g', skip: 240 })).n, 10);
|
||||
eq('23 count skip and limit', (await db.command({ count: 'g', skip: 245, limit: 10 })).n, 5);
|
||||
eq('23 count skip past the end', (await db.command({ count: 'g', skip: 1000 })).n, 0);
|
||||
eq('23 count with a query and limit', (await db.command({ count: 'g', query: { g: 0 }, limit: 3 })).n, 3);
|
||||
eq('23 driver countDocuments limit', await col.countDocuments({}, { limit: 7 }), 7);
|
||||
|
||||
// 24. A batch is capped by bytes as well as by documents, so a large-document
|
||||
// result splits instead of building a reply past the advertised message
|
||||
// size. 40 documents of ~1 MiB cannot all fit one 16 MiB batch.
|
||||
const big = db.collection('big');
|
||||
await big.deleteMany({});
|
||||
const pad = 'p'.repeat(1024 * 1024 - 64);
|
||||
for (let i = 0; i < 40; i++) await big.insertOne({ _id: i + 1, pad });
|
||||
r = await db.command({ find: 'big', filter: {}, batchSize: 40 });
|
||||
check('24 the byte cap split the batch', r.cursor.firstBatch.length >= 1 && r.cursor.firstBatch.length <= 16, r.cursor.firstBatch.length);
|
||||
check('24 and left the cursor open', r.cursor.id > 0);
|
||||
eq('24 the whole result still drains', (await big.find({}).batchSize(40).toArray()).length, 40);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Phase D — expiry, capacity, and what a restart does
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
async function phaseD(db) {
|
||||
const col = db.collection('e');
|
||||
await col.deleteMany({});
|
||||
await col.insertMany([...Array(50)].map((_, i) => ({ _id: i + 1 })));
|
||||
|
||||
// 25. An idle cursor is reaped; noCursorTimeout exempts one from that.
|
||||
const perishable = (await db.command({ find: 'e', filter: {}, batchSize: 2 })).cursor.id;
|
||||
const immortal = (await db.command({ find: 'e', filter: {}, batchSize: 2, noCursorTimeout: true })).cursor.id;
|
||||
await sleep(300);
|
||||
eq('25 before the timeout it is alive', await codeOf(() => db.command({ getMore: perishable, collection: 'e', batchSize: 1 })), 0);
|
||||
await sleep(2500);
|
||||
eq('25 an idle cursor is reaped', await codeOf(() => db.command({ getMore: perishable, collection: 'e', batchSize: 1 })), 43);
|
||||
eq('25 noCursorTimeout survives', await codeOf(() => db.command({ getMore: immortal, collection: 'e', batchSize: 1 })), 0);
|
||||
const k = await db.command({ killCursors: 'e', cursors: [immortal] });
|
||||
eq('25 but is still killable', k.cursorsKilled.map(String), [String(immortal)]);
|
||||
|
||||
// 26. A full registry evicts the least recently used cursor rather than
|
||||
// refusing the new one. The victim sees the same 43 an idle timeout gives,
|
||||
// which every driver already handles.
|
||||
const ids = [];
|
||||
for (let i = 0; i < 5; i++) ids.push((await db.command({ find: 'e', filter: {}, batchSize: 1 })).cursor.id);
|
||||
eq('26 the oldest was evicted', await codeOf(() => db.command({ getMore: ids[0], collection: 'e', batchSize: 1 })), 43);
|
||||
const alive = [];
|
||||
for (const id of ids.slice(1)) alive.push(await codeOf(() => db.command({ getMore: id, collection: 'e', batchSize: 1 })));
|
||||
eq('26 the newest four are alive', alive, [0, 0, 0, 0]);
|
||||
}
|
||||
|
||||
async function phaseE(db, staleId) {
|
||||
// 27. Cursors do not survive a restart, and a stale id must be a clean 43 —
|
||||
// not a hang, and not an empty batch claiming the result ended.
|
||||
eq('27 a cursor from before the restart is 43', await codeOf(() => db.command({ getMore: staleId, collection: 'e', batchSize: 1 })), 43);
|
||||
const fresh = await db.command({ find: 'e', filter: {}, batchSize: 2 });
|
||||
check('27 and new cursors work after a restart', fresh.cursor.id > 0);
|
||||
}
|
||||
|
||||
async function main() {
|
||||
// The same guard e2e6.js and big.js carry: without it a missing binary
|
||||
// surfaces as a generic spawn error instead of saying what to do about it.
|
||||
if (!fs.existsSync(BIN)) {
|
||||
console.error(`E2E7_FAIL server binary not found: ${BIN}\n run: zig build`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Phase A-C on default cursor flags.
|
||||
await startServer(['--ttl-sweep-secs', '0', '--compact-threshold', '1m'], true);
|
||||
let client = new MongoClient(URL);
|
||||
await client.connect();
|
||||
let db = client.db('e2e7');
|
||||
console.log('phase A: batching, lifecycle, errors');
|
||||
await phaseA(db);
|
||||
console.log('phase B: streaming cursors across writes');
|
||||
await phaseB(db);
|
||||
console.log('phase C: aggregate, listings, count');
|
||||
await phaseC(db);
|
||||
await client.close();
|
||||
await stopServer('SIGTERM');
|
||||
|
||||
// Phase D needs a short timeout and a tiny registry.
|
||||
console.log('phase D: idle expiry and registry capacity');
|
||||
await startServer(
|
||||
['--ttl-sweep-secs', '0', '--cursor-timeout-ms', '800', '--cursor-sweep-secs', '1', '--max-open-cursors', '4'],
|
||||
true,
|
||||
);
|
||||
client = new MongoClient(URL);
|
||||
await client.connect();
|
||||
db = client.db('e2e7');
|
||||
await phaseD(db);
|
||||
const staleId = (await db.command({ find: 'e', filter: {}, batchSize: 1 })).cursor.id;
|
||||
await client.close();
|
||||
await stopServer('SIGTERM');
|
||||
|
||||
// Phase E: the same database, a new process.
|
||||
console.log('phase E: a cursor does not survive a restart');
|
||||
await startServer(['--ttl-sweep-secs', '0'], false);
|
||||
client = new MongoClient(URL);
|
||||
await client.connect();
|
||||
await phaseE(client.db('e2e7'), staleId);
|
||||
await client.close();
|
||||
|
||||
if (process.env.E2E7_KEEP !== '1') {
|
||||
fs.rmSync(DBFILE, { force: true });
|
||||
fs.rmSync(DBFILE + '.data', { force: true });
|
||||
}
|
||||
await stopServer('SIGTERM');
|
||||
|
||||
const failed = results.filter((r) => !r.ok);
|
||||
console.log(`\n${results.length - failed.length}/${results.length} checks passed`);
|
||||
if (failed.length) {
|
||||
console.log('FAILED:', failed.map((f) => f.name).join(', '));
|
||||
console.log('--- server log tail ---');
|
||||
console.log(serverLog.split('\n').slice(-30).join('\n'));
|
||||
process.exit(1);
|
||||
}
|
||||
console.log('E2E7_OK');
|
||||
}
|
||||
|
||||
main().catch((e) => {
|
||||
console.error('E2E7_FAIL', e);
|
||||
console.log('--- server log tail ---');
|
||||
console.log(serverLog.split('\n').slice(-40).join('\n'));
|
||||
process.exit(1);
|
||||
});
|
||||
Reference in New Issue
Block a user