storage/db: XxHash3 record integrity, garbage-ratio compaction
Two streams of work land together: they are interleaved in storage.zig
and db.zig and only build as a unit.
Already in the working tree before this session:
- ReleaseFast as the default zig build (Debug was 10-200x slower)
- group commit: one fsync per write command instead of per document
- plan_id returned a pointer to a stack temporary; ReleaseFast read
garbage and silently broke findOne({_id: ObjectId})
- perf suite: big.js, compare.js, compare-run.sh, e2e6.js
Phase 1 performance work:
Record integrity hash CRC32 -> XxHash3. std.hash.Crc32 is the
table-driven byte-at-a-time Crc32IsoHdlc, measured at 408 MB/s against
XxHash3's 31 GB/s: 38us versus 0.5us on a 16 KiB document, which was
about two thirds of the entire bulk-insert cost. The record header
grows from u32 crc to u64 hash (header_len 20 -> 24), a breaking
format change. Bulk insert 260 -> 700 MB/s, reopen 1.1 -> 0.5s.
Compaction fsynced once per live document, because Log.open leaves
defer_sync false and compact never set it. It now issues one sync for
the whole rewrite, before the rename that publishes it.
Compaction triggers on the share of the log that is garbage
(live_docs/dead_docs, maintained at evict_doc, the single point where
a document dies) rather than on bytes appended. A fixed byte count is
wrong in both directions: a 1 GB bulk load holds no garbage at all yet
would compact ~64 times under the 16 MiB default, rewriting 1 GB each
time, while a small collection rewritten in place accumulates garbage
indefinitely without ever reaching the count. Pure inserts now never
compact, and the file stays near 1.25x the live data. Bulk load at the
default threshold: 41.6 -> 702.7 MB/s.
remove() never called maybe_compact, so a delete-heavy workload grew
the log without bound.
e2e6's compaction check required the file to bloat past 30 MiB before
being reclaimed, which encoded the old policy and failed on strictly
better behaviour (ends at 15.8 MB against ~12 MB live, was ~30 MB). It
now asserts the file ends near the live size and peaked well above it,
which does not depend on when the trigger fires. Sampling interval 50
-> 10ms: the operations now finish inside the old window.
Verified: 70 unit tests under both ReleaseFast and ReleaseSafe, e2e
29/16/17/3/2 checks, the crash-a/kill -9/crash-b pair, and e2e6 72/72
across three consecutive runs.
This commit is contained in:
@@ -1,7 +1,8 @@
|
||||
# End-to-end tests with the official MongoDB Node.js driver
|
||||
|
||||
These exercise mongo-lite from a real driver over TCP: full CRUD, query
|
||||
operators, aggregation, error codes, concurrent clients, and crash recovery.
|
||||
operators, aggregation, error codes, concurrent clients, crash recovery, and
|
||||
the whole lifecycle including server restarts.
|
||||
|
||||
## Setup
|
||||
|
||||
@@ -13,7 +14,7 @@ npm install mongodb
|
||||
|
||||
## Run
|
||||
|
||||
Start the server, then run the suites against it (defaults to port 27020):
|
||||
Most suites expect a server running on port 27020:
|
||||
|
||||
```sh
|
||||
zig build
|
||||
@@ -28,8 +29,17 @@ node tests/e2e/e2e4.js # TTL indexes: expiry + rejected specs (15 checks
|
||||
```
|
||||
|
||||
`e2e4.js` needs the server started with `--ttl-sweep-secs 1` (the default is
|
||||
60 seconds, which is longer than the suite waits); the other suites do not
|
||||
care about the flag.
|
||||
60 seconds); the other suites do not care about the flag.
|
||||
|
||||
`e2e6.js` is the full-lifecycle suite and is self-contained: it spawns its
|
||||
own server on port 27220 with a fresh log, runs the whole feature surface,
|
||||
restarts the server twice (graceful SIGTERM, then kill -9 mid-write) and
|
||||
verifies everything survived:
|
||||
|
||||
```sh
|
||||
node tests/e2e/e2e6.js # 73 checks, ~15 s, needs no running server
|
||||
E2E6_PORT=27300 node tests/e2e/e2e6.js # different port if 27220 is taken
|
||||
```
|
||||
|
||||
Rebuild with `zig build` after any change under `src/` before restarting the
|
||||
server: `zig build test` compiles the test binary only and leaves
|
||||
@@ -38,3 +48,52 @@ rules and report failures that the source no longer explains.
|
||||
|
||||
`e2e2.js concurrent` is safe to repeat against a running server (it drops its
|
||||
collection first); `crash-a`/`crash-b` are two halves of one scenario.
|
||||
|
||||
## Multi-GB collections: `big.js`
|
||||
|
||||
`big.js` is a load harness, not a pass/fail suite: it spawns a server, bulk
|
||||
loads up to ~5 GB, and reports insert throughput, the compaction behavior,
|
||||
server RSS, per-operation latencies, reopen (replay) time, and kill -9
|
||||
durability.
|
||||
|
||||
```sh
|
||||
node tests/e2e/big.js --quick # 268 MB smoke run
|
||||
node tests/e2e/big.js --size 5g --doc-size 128k --oid --batch 200 \
|
||||
--compact-threshold 2g # ~5 GB, 40k docs
|
||||
```
|
||||
|
||||
Options: `--size`/`--doc-size`/`--batch` (k/m/g suffixes), `--oid`
|
||||
(ObjectId `_id`s — see below), `--index <field>` (secondary index before
|
||||
loading), `--compact-threshold <bytes>` (passed to the server),
|
||||
`--port`, `--keep` (keep the db file).
|
||||
|
||||
Measured behavior (all documented in the top-level README):
|
||||
|
||||
- **Build in ReleaseFast** — `zig build` defaults to it; a Debug server is
|
||||
10-200x slower on every path.
|
||||
- **Insert throughput collapses under the default 16 MiB compaction
|
||||
threshold**: every ~16 MB of writes rewrites the whole log with one fsync
|
||||
per record (O(n²) total). With `--compact-threshold 2g` the rate stays
|
||||
flat (hundreds of MB/s at 128 KB docs in ReleaseFast). Raise the
|
||||
threshold for bulk loads.
|
||||
- **`findOne({_id})` is O(1) only for ObjectId `_id`s.** Integer `_id`s are
|
||||
serialization-ambiguous (int32/int64/double compare equal but hash
|
||||
differently), so the docs-map fast path is skipped and every lookup is a
|
||||
full scan. Use the driver's default ObjectId ids on big collections.
|
||||
- **The engine holds everything in RAM**: ~1-1.2x the data size at 128 KB
|
||||
docs (more at 16 KB docs, where per-document arena overhead dominates).
|
||||
A 5 GB collection needs roughly 6-7 GB of RAM.
|
||||
- Reopen of a 5 GB log replays in ~10 s (ReleaseFast); every committed
|
||||
write survives kill -9.
|
||||
|
||||
## Comparing against real MongoDB: `compare.js` + `compare-run.sh`
|
||||
|
||||
```sh
|
||||
bash tests/e2e/compare-run.sh [size] [doc-size] # e.g. 1g 16k
|
||||
```
|
||||
|
||||
Starts mongod (`brew install mongodb-community`) on :27018 and mongo-lite
|
||||
on :27019, runs the same driver workload against each (durable writes:
|
||||
mongo-lite fsyncs per command, mongod runs with `j: true`), measures kill -9
|
||||
reopen for both, and prints a side-by-side table. `compare.js` alone runs
|
||||
one side (see its `--help`-style header comment).
|
||||
|
||||
347
tests/e2e/big.js
Normal file
347
tests/e2e/big.js
Normal file
@@ -0,0 +1,347 @@
|
||||
// Big-collection harness: how mongo-lite behaves with multi-GB collections.
|
||||
//
|
||||
// Spawns its own server, bulk-inserts up to ~5 GB of documents, measures
|
||||
// insert throughput, log/compaction behavior and server RSS, benchmarks
|
||||
// find/count/update/delete against the full dataset, then kills the server
|
||||
// and measures reopen (replay) time and crash durability.
|
||||
//
|
||||
// node tests/e2e/big.js [options]
|
||||
// --size <n> target collection size; k/m/g suffixes (default 5g)
|
||||
// --doc-size <n> approximate bytes per document (default 32k)
|
||||
// --batch <n> docs per insertMany call (default 500)
|
||||
// --index <f> create a secondary index on field f *before* inserting
|
||||
// (entry insert is O(n), so this makes the load quadratic)
|
||||
// --oid use ObjectId _ids (driver-generated): O(1) _id lookups.
|
||||
// Without it, int _ids fall back to a full scan (the _id
|
||||
// fast path is skipped for serialization-ambiguous
|
||||
// numeric classes), so findOne({_id}) costs a scan.
|
||||
// --compact-threshold <n>
|
||||
// pass through to the server: log bytes between
|
||||
// compactions (default 16m). Raise for bulk loads.
|
||||
// --port <n> server port (default 27221)
|
||||
// --keep keep the db file after the run
|
||||
// --quick tiny run (256m, 16k docs)
|
||||
//
|
||||
// Env: ML_BIN server binary (default ../../zig-out/bin/mongo-lite)
|
||||
// BIG_DB db file path (default .zig-cache/big.log)
|
||||
const { MongoClient } = require('mongodb');
|
||||
const { spawn } = require('child_process');
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
|
||||
const BIN = process.env.ML_BIN || path.resolve(__dirname, '../../zig-out/bin/mongo-lite');
|
||||
const PORT = Number(process.env.BIG_PORT || 27221);
|
||||
const DBFILE = process.env.BIG_DB || path.resolve(__dirname, '../../.zig-cache/big.log');
|
||||
const URL = `mongodb://127.0.0.1:${PORT}`;
|
||||
|
||||
function parseSize(s) {
|
||||
const m = /^(\d+(?:\.\d+)?)([kmgt]?)$/i.exec(String(s).trim());
|
||||
if (!m) throw new Error(`bad size '${s}'`);
|
||||
const mult = { '': 1, k: 1 << 10, m: 1 << 20, g: 1 << 30, t: 1 << 40 }[m[2].toLowerCase()];
|
||||
return Math.round(parseFloat(m[1]) * mult);
|
||||
}
|
||||
|
||||
let opt = { size: '5g', docSize: '32k', batch: 500, index: null, port: PORT, keep: false, oid: false, compactThreshold: null };
|
||||
for (let i = 2; i < process.argv.length; i++) {
|
||||
const a = process.argv[i];
|
||||
if (a === '--quick') { opt.size = '256m'; opt.docSize = '16k'; }
|
||||
else if (a === '--keep') opt.keep = true;
|
||||
else if (a === '--oid') opt.oid = true;
|
||||
else if (a === '--index') opt.index = process.argv[++i];
|
||||
else if (a.startsWith('--size=')) opt.size = a.slice(7);
|
||||
else if (a.startsWith('--doc-size=')) opt.docSize = a.slice(11);
|
||||
else if (a.startsWith('--batch=')) opt.batch = Number(a.slice(8));
|
||||
else if (a.startsWith('--port=')) opt.port = Number(a.slice(7));
|
||||
else if (a.startsWith('--compact-threshold=')) opt.compactThreshold = a.slice(20);
|
||||
else if (a === '--size') opt.size = process.argv[++i];
|
||||
else if (a === '--doc-size') opt.docSize = process.argv[++i];
|
||||
else if (a === '--batch') opt.batch = Number(process.argv[++i]);
|
||||
else if (a === '--compact-threshold') opt.compactThreshold = process.argv[++i];
|
||||
else { console.error(`unknown option ${a}`); process.exit(2); }
|
||||
}
|
||||
|
||||
const SIZE = parseSize(opt.size);
|
||||
const DOC_SIZE = parseSize(opt.docSize);
|
||||
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
|
||||
|
||||
let server = null;
|
||||
let serverLog = '';
|
||||
let serverDead = false;
|
||||
function cleanup() {
|
||||
if (server && !serverDead) {
|
||||
try { server.kill('SIGKILL'); } catch {}
|
||||
}
|
||||
}
|
||||
process.on('exit', cleanup);
|
||||
process.on('SIGINT', () => { cleanup(); process.exit(130); });
|
||||
process.on('SIGTERM', () => { cleanup(); process.exit(143); });
|
||||
function startServer() {
|
||||
return new Promise((resolve, reject) => {
|
||||
const t0 = Date.now();
|
||||
serverDead = false;
|
||||
const args = ['--port', String(opt.port), '--db', DBFILE];
|
||||
if (opt.compactThreshold) args.push('--compact-threshold', opt.compactThreshold);
|
||||
server = spawn(BIN, args, { stdio: ['ignore', 'pipe', 'pipe'] });
|
||||
server.stdout.on('data', (d) => (serverLog += d));
|
||||
server.stderr.on('data', (d) => (serverLog += d));
|
||||
server.on('error', (e) => reject(new Error(`cannot start ${BIN}: ${e.message}`)));
|
||||
server.on('exit', (code, sig) => {
|
||||
serverDead = true;
|
||||
if (code !== null && sig === null) serverLog += `\n[child exited rc=${code}]`;
|
||||
});
|
||||
const deadline = Date.now() + 120000;
|
||||
(async () => {
|
||||
while (Date.now() < deadline) {
|
||||
if (serverDead) {
|
||||
reject(new Error(`server child exited during start (port ${opt.port} busy?)\n${serverLog}`));
|
||||
return;
|
||||
}
|
||||
const c = new MongoClient(URL, { serverSelectionTimeoutMS: 1000 });
|
||||
try {
|
||||
await c.connect();
|
||||
await c.db('admin').command({ ping: 1 });
|
||||
await c.close();
|
||||
return resolve(Date.now() - t0);
|
||||
} catch {
|
||||
try { await c.close(); } catch {}
|
||||
await sleep(100);
|
||||
}
|
||||
}
|
||||
reject(new Error(`server did not come up on :${opt.port}\n${serverLog}`));
|
||||
})();
|
||||
});
|
||||
}
|
||||
async function stopServer(sig = 'SIGKILL') {
|
||||
if (!server) return;
|
||||
const exited = new Promise((r) => server.once('exit', r));
|
||||
server.kill(sig);
|
||||
await Promise.race([exited, sleep(5000)]);
|
||||
serverDead = true;
|
||||
server = null;
|
||||
}
|
||||
async function rssMB() {
|
||||
if (!server) return 0;
|
||||
try {
|
||||
const out = (await new Promise((r) => require('child_process').exec(`ps -o rss= -p ${server.pid}`, (e, so) => r(so || '')))).trim();
|
||||
return Math.round(Number(out) / 1024);
|
||||
} catch { return 0; }
|
||||
}
|
||||
|
||||
const report = [];
|
||||
function row(label, value) { report.push([label, value]); console.log(` ${String(label).padEnd(46)} ${value}`); }
|
||||
|
||||
async function main() {
|
||||
if (!fs.existsSync(BIN)) {
|
||||
console.error(`server binary not found at ${BIN} — run \`zig build\` first`);
|
||||
process.exit(1);
|
||||
}
|
||||
fs.rmSync(DBFILE, { force: true });
|
||||
const fmt = (n) => (n >= 1e9 ? (n / 1e9).toFixed(2) + ' GB' : n >= 1e6 ? (n / 1e6).toFixed(1) + ' MB' : n >= 1e3 ? (n / 1e3).toFixed(1) + ' KB' : n + ' B');
|
||||
console.log(`mongo-lite big-collection harness`);
|
||||
console.log(` size ${fmt(SIZE)} · doc ~${fmt(DOC_SIZE)} · batch ${opt.batch} · index ${opt.index || 'none'} · ids ${opt.oid ? 'ObjectId' : 'int'} · compact-threshold ${opt.compactThreshold || '16m'} · db ${DBFILE}`);
|
||||
|
||||
console.log('\n== server start (fresh log) ==');
|
||||
const openMs = await startServer();
|
||||
row('open + first ping (fresh log)', `${openMs} ms`);
|
||||
|
||||
const client = new MongoClient(URL, { serverSelectionTimeoutMS: 10000 });
|
||||
await client.connect();
|
||||
const db = client.db('big');
|
||||
const coll = db.collection('items');
|
||||
await coll.drop().catch(() => {});
|
||||
|
||||
if (opt.index) {
|
||||
const t0 = Date.now();
|
||||
await coll.createIndex({ [opt.index]: 1 });
|
||||
row(`createIndex({${opt.index}: 1}) on empty coll`, `${Date.now() - t0} ms`);
|
||||
}
|
||||
|
||||
// ---- insert ------------------------------------------------------------
|
||||
console.log('\n== insert ==');
|
||||
const nDocs = Math.max(1, Math.ceil(SIZE / DOC_SIZE));
|
||||
const payloadLen = Math.max(1, DOC_SIZE - 130); // bson overhead for _id/k/p/ts/payload
|
||||
const payload = 'x'.repeat(payloadLen);
|
||||
const { ObjectId } = require('mongodb');
|
||||
const t0 = Date.now();
|
||||
const logSamples = [{ t: 0, size: fs.existsSync(DBFILE) ? fs.statSync(DBFILE).size : 0, rss: 0 }];
|
||||
const sampler = setInterval(async () => {
|
||||
logSamples.push({ t: Date.now() - t0, size: fs.existsSync(DBFILE) ? fs.statSync(DBFILE).size : 0, rss: await rssMB() });
|
||||
}, 2000);
|
||||
|
||||
// Rate curve: avg MB/s per progress chunk, to show how throughput changes
|
||||
// as the log grows (compaction rewrites + fsync-per-record dominate).
|
||||
const rates = [];
|
||||
let lastProgressT = t0;
|
||||
let lastProgressDocs = 0;
|
||||
let midDocId = null; // ObjectId of the mid doc (oid mode), captured at insert
|
||||
let lastDocId = null;
|
||||
let inserted = 0;
|
||||
const midIndex = Math.floor(nDocs / 2);
|
||||
try {
|
||||
while (inserted < nDocs) {
|
||||
const n = Math.min(opt.batch, nDocs - inserted);
|
||||
const docs = new Array(n);
|
||||
for (let i = 0; i < n; i++) {
|
||||
const id = inserted + i + 1;
|
||||
const doc = opt.oid
|
||||
? { _id: new ObjectId(), k: id % 1000, p: id % 5000, ts: new Date(Date.UTC(2024, 0, 1) + id * 1000), payload }
|
||||
: { _id: id, k: id % 1000, p: id % 5000, ts: new Date(Date.UTC(2024, 0, 1) + id * 1000), payload };
|
||||
if (id === midIndex) midDocId = doc._id;
|
||||
if (id === nDocs) lastDocId = doc._id;
|
||||
docs[i] = doc;
|
||||
}
|
||||
await coll.insertMany(docs, { ordered: false });
|
||||
inserted += n;
|
||||
if (inserted % Math.max(1000, Math.floor(nDocs / 20)) < n) {
|
||||
const dt = (Date.now() - t0) / 1000;
|
||||
const mb = inserted * DOC_SIZE / 1e6;
|
||||
const chunkMb = (inserted - lastProgressDocs) * DOC_SIZE / 1e6;
|
||||
const chunkT = (Date.now() - lastProgressT) / 1000;
|
||||
rates.push(+(chunkMb / chunkT).toFixed(1));
|
||||
lastProgressT = Date.now();
|
||||
lastProgressDocs = inserted;
|
||||
console.log(` ${inserted.toLocaleString()} docs · ${fmt(inserted * DOC_SIZE)} · ${(mb / dt).toFixed(1)} MB/s avg`);
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
clearInterval(sampler);
|
||||
}
|
||||
const insertMs = Date.now() - t0;
|
||||
const bytes = inserted * DOC_SIZE;
|
||||
row('docs inserted', inserted.toLocaleString());
|
||||
row('approx bytes', fmt(bytes));
|
||||
row('wall time', `${(insertMs / 1000).toFixed(1)} s`);
|
||||
row('throughput', `${(bytes / 1e6 / (insertMs / 1000)).toFixed(1)} MB/s (${(inserted / (insertMs / 1000)).toFixed(0)} docs/s)`);
|
||||
row('rate curve (MB/s per chunk)', rates.join(' → ') || 'n/a');
|
||||
|
||||
const compactions = logSamples.filter((s, i) => i > 0 && s.size < logSamples[i - 1].size - 2 * 1024 * 1024).length;
|
||||
const sizes = logSamples.map((s) => s.size);
|
||||
row('log file size (min → final)', `${fmt(Math.min(...sizes))} → ${fmt(sizes[sizes.length - 1])}`);
|
||||
row('compaction events observed', compactions, '(log shrank by >2MB between samples)');
|
||||
const peakRss = Math.max(...logSamples.map((s) => s.rss));
|
||||
row('peak server RSS', `${peakRss} MB`, '(in-memory engine: docs live in RAM)');
|
||||
|
||||
// ---- find / read -------------------------------------------------------
|
||||
console.log('\n== find / read on full dataset ==');
|
||||
const bench = async (label, fn, min = 1) => {
|
||||
const a = Date.now();
|
||||
const res = await fn();
|
||||
const ms = Date.now() - a;
|
||||
row(`${label}`, `${ms < 1000 ? ms + ' ms' : (ms / 1000).toFixed(2) + ' s'}${res !== undefined ? ` (${res})` : ''}`);
|
||||
return ms;
|
||||
};
|
||||
await bench('countDocuments({})', async () => {
|
||||
const n = await coll.countDocuments({});
|
||||
if (n !== inserted) throw new Error(`count ${n} != ${inserted}`);
|
||||
return `${n.toLocaleString()} docs`;
|
||||
});
|
||||
await bench('findOne({_id: mid}) — ' + (opt.oid ? 'docs-map fast path' : 'scan (int _id: fast path skipped)'), async () => {
|
||||
const d = await coll.findOne({ _id: opt.oid ? midDocId : midIndex });
|
||||
if (!d) throw new Error('miss');
|
||||
});
|
||||
await bench('findOne({_id: last})', async () => {
|
||||
const d = await coll.findOne({ _id: opt.oid ? lastDocId : nDocs });
|
||||
if (!d) throw new Error('miss');
|
||||
});
|
||||
if (opt.index) {
|
||||
await bench(`find({${opt.index}: 4242}).count() — via index`, async () => {
|
||||
const n = await coll.countDocuments({ k: 4242 });
|
||||
if (n < 1) throw new Error('no hits');
|
||||
return `${n} hits`;
|
||||
});
|
||||
}
|
||||
await bench('find({k: 4242}).count() — scan', async () => {
|
||||
const n = await coll.countDocuments({ k: 4242 });
|
||||
return `${n} hits`;
|
||||
});
|
||||
await bench('find({p: {$gte, $lt}}).count() — range scan', async () => {
|
||||
const lo = 1000, hi = 2000;
|
||||
const n = await coll.countDocuments({ p: { $gte: lo, $lt: hi } });
|
||||
return `${n} hits`;
|
||||
});
|
||||
await bench('find({}).sort({_id:-1}).limit(20) — full scan + sort', async () => {
|
||||
const docs = await coll.find({}).sort({ _id: -1 }).limit(20).toArray();
|
||||
if (docs.length !== 20) throw new Error('bad page');
|
||||
});
|
||||
await bench('find({}, {proj: _id,k,p}).limit(500) — page', async () => {
|
||||
const docs = await coll.find({}, { projection: { payload: 0 } }).limit(500).toArray();
|
||||
if (docs.length !== 500) throw new Error('short page');
|
||||
});
|
||||
|
||||
// ---- write ops against the full dataset ---------------------------------
|
||||
console.log('\n== point write ops ==');
|
||||
const midId = opt.oid ? midDocId : midIndex;
|
||||
await bench('updateOne({_id: mid}, {$set}) — 1 fsync', async () => {
|
||||
const r = await coll.updateOne({ _id: midId }, { $set: { touch: Date.now() } });
|
||||
if (r.modifiedCount !== 1) throw new Error('miss');
|
||||
});
|
||||
await bench('updateMany({k: 7}, {$inc}) — ~N/1000 fsyncs', async () => {
|
||||
const r = await coll.updateMany({ k: 7 }, { $inc: { hits: 1 } });
|
||||
return `${r.modifiedCount} modified`;
|
||||
});
|
||||
await bench('deleteOne({_id: mid}) + re-insert', async () => {
|
||||
await coll.deleteOne({ _id: midId });
|
||||
if (opt.oid) {
|
||||
await coll.insertOne({ _id: new ObjectId(), k: midIndex % 1000, p: midIndex % 5000, payload });
|
||||
} else {
|
||||
await coll.insertOne({ _id: midIndex, k: midIndex % 1000, p: midIndex % 5000, payload });
|
||||
}
|
||||
});
|
||||
await bench('aggregate $group by k', async () => {
|
||||
const out = await coll.aggregate([{ $group: { _id: '$k', n: { $sum: 1 } } }]).toArray();
|
||||
return `${out.length} groups`;
|
||||
});
|
||||
|
||||
// ---- reopen (replay) ----------------------------------------------------
|
||||
console.log('\n== durability ==');
|
||||
await client.close();
|
||||
await stopServer('SIGKILL');
|
||||
const reopenMs = await startServer();
|
||||
row('kill -9 then reopen (replay of full log)', `${(reopenMs / 1000).toFixed(1)} s`);
|
||||
const c2 = new MongoClient(URL, { serverSelectionTimeoutMS: 10000 });
|
||||
await c2.connect();
|
||||
const db2 = c2.db('big');
|
||||
const coll2 = db2.collection('items');
|
||||
const afterRestart = await coll2.countDocuments({});
|
||||
row('count after restart', `${afterRestart.toLocaleString()} (${afterRestart === inserted ? 'OK' : 'MISMATCH!'})`);
|
||||
const spot = await coll2.findOne({ _id: opt.oid ? lastDocId : nDocs });
|
||||
row('last doc intact after restart', spot ? `payload ${spot.payload.length}B` : 'MISSING!');
|
||||
if (afterRestart !== inserted || !spot) throw new Error('durability check failed');
|
||||
|
||||
// Crash-durability: every write is fsynced before it is acknowledged, so a
|
||||
// kill -9 right after an insert must not lose it.
|
||||
const crash = db2.collection('crash');
|
||||
await crash.drop().catch(() => {});
|
||||
let committed = 0;
|
||||
for (let i = 1; i <= 200; i++) {
|
||||
await crash.insertOne({ _id: i, seq: i });
|
||||
committed = i;
|
||||
}
|
||||
await c2.close();
|
||||
await stopServer('SIGKILL');
|
||||
await startServer();
|
||||
const c3 = new MongoClient(URL, { serverSelectionTimeoutMS: 10000 });
|
||||
await c3.connect();
|
||||
const db3 = c3.db('big');
|
||||
const crashN = await db3.collection('crash').countDocuments({});
|
||||
row('kill -9 after 200 committed writes', `${crashN}/200 survived (${crashN === 200 ? 'OK' : 'MISMATCH!'})`);
|
||||
await c3.close();
|
||||
|
||||
if (!opt.keep) fs.rmSync(DBFILE, { force: true });
|
||||
await stopServer('SIGKILL');
|
||||
|
||||
console.log('\n== summary ==');
|
||||
console.log(` mongo-lite handles a ${fmt(bytes)} collection fully in RAM (RSS ${peakRss} MB)`);
|
||||
console.log(` insert: ${(bytes / 1e6 / (insertMs / 1000)).toFixed(1)} MB/s — fsync per write is by design (crash safety)`);
|
||||
if (compactions > 0) {
|
||||
console.log(` ${compactions} compaction rewrites observed: every 16MB of writes rewrites the whole log — for multi-GB loads the cumulative rewrite traffic dominates`);
|
||||
}
|
||||
console.log('BIG_OK');
|
||||
}
|
||||
|
||||
main().catch((e) => {
|
||||
console.error('BIG_FAIL', e);
|
||||
console.log('--- server log tail ---');
|
||||
console.log(serverLog.split('\n').slice(-40).join('\n'));
|
||||
process.exit(1);
|
||||
});
|
||||
119
tests/e2e/compare-run.sh
Normal file
119
tests/e2e/compare-run.sh
Normal file
@@ -0,0 +1,119 @@
|
||||
#!/bin/bash
|
||||
# Compare mongo-lite against a real MongoDB with the same workload, same driver.
|
||||
#
|
||||
# bash tests/e2e/compare-run.sh [size] [doc-size]
|
||||
# (defaults: 1g, 16k)
|
||||
#
|
||||
# Starts mongod on :27018 and mongo-lite on :27019, runs compare.js against
|
||||
# each (durable writes: mongo-lite fsyncs per doc, mongod ack'd with j:true),
|
||||
# measures kill -9 reopen time for both, and prints a side-by-side table.
|
||||
set -u
|
||||
cd "$(dirname "$0")/../.."
|
||||
SIZE="${1:-1g}"
|
||||
DOC="${2:-16k}"
|
||||
echo "comparing mongo-lite vs mongodb — dataset ${SIZE}, docs ~${DOC}"
|
||||
|
||||
CMPDIR=/tmp/mongo-cmp
|
||||
mkdir -p "$CMPDIR/mongod"
|
||||
ML_LOG="$CMPDIR/ml.log"
|
||||
ML_OUT="$CMPDIR/ml-srv.out"
|
||||
MD_OUT="$CMPDIR/md-srv.out"
|
||||
ML_PORT=27019
|
||||
MD_PORT=27018
|
||||
rm -f "$ML_LOG"
|
||||
rm -rf "$CMPDIR/mongod" && mkdir -p "$CMPDIR/mongod"
|
||||
|
||||
# ---- MongoDB -------------------------------------------------------------
|
||||
echo; echo "### mongod (MongoDB $(mongod --version | grep -oE 'v[0-9.]+' | head -1))"
|
||||
mongod --dbpath "$CMPDIR/mongod" --port $MD_PORT --bind_ip 127.0.0.1 \
|
||||
--quiet >"$MD_OUT" 2>&1 &
|
||||
MD_PID=$!
|
||||
sleep 2
|
||||
node tests/e2e/compare.js --url "mongodb://127.0.0.1:$MD_PORT" --label mongodb --size "$SIZE" --doc-size "$DOC" \
|
||||
> "$CMPDIR/mongo-report.txt" 2>&1 || { echo "mongodb bench failed:"; tail -5 "$CMPDIR/mongo-report.txt"; }
|
||||
MD_RSS=$(ps -o rss= -p $MD_PID | awk '{printf "%.0f", $1/1024}')
|
||||
MD_DISK=$(du -sm "$CMPDIR/mongod" | awk '{print $1}')
|
||||
|
||||
echo; echo "### mongod kill -9 + reopen"
|
||||
kill -9 $MD_PID; wait $MD_PID 2>/dev/null
|
||||
MD_REOPEN=$(cd tests/e2e && node -e '
|
||||
const { spawn } = require("child_process");
|
||||
const { MongoClient } = require("mongodb");
|
||||
const t0 = Date.now();
|
||||
const p = spawn("mongod", ["--dbpath","/tmp/mongo-cmp/mongod","--port","27018","--bind_ip","127.0.0.1","--quiet"], {stdio:"ignore"});
|
||||
const poll = async () => {
|
||||
const c = new MongoClient("mongodb://127.0.0.1:27018", {serverSelectionTimeoutMS: 800});
|
||||
try { await c.connect(); await c.db("admin").command({ping:1}); await c.close(); p.kill("SIGKILL"); console.log(((Date.now()-t0)/1000).toFixed(1)); }
|
||||
catch { try { await c.close(); } catch {}; setTimeout(poll, 200); }
|
||||
};
|
||||
setTimeout(poll, 300);
|
||||
')
|
||||
kill -9 $MD_PID 2>/dev/null
|
||||
|
||||
# ---- mongo-lite ----------------------------------------------------------
|
||||
echo; echo "### mongo-lite (recommended config: --compact-threshold 1g)"
|
||||
# Debug is ~10-200x slower (see the README's perf section) — the comparison
|
||||
# must use the optimized build.
|
||||
zig build -Doptimize=ReleaseFast 2>&1 | grep -c "^error" | grep -q "^0" || { echo "build failed"; exit 1; }
|
||||
./zig-out/bin/mongo-lite --port $ML_PORT --db "$ML_LOG" --compact-threshold 1g >"$ML_OUT" 2>&1 &
|
||||
ML_PID=$!
|
||||
sleep 1
|
||||
node tests/e2e/compare.js --url "mongodb://127.0.0.1:$ML_PORT" --label mongo-lite --size "$SIZE" --doc-size "$DOC" \
|
||||
> "$CMPDIR/ml-report.txt" 2>&1 || { echo "mongo-lite bench failed:"; tail -5 "$CMPDIR/ml-report.txt"; }
|
||||
ML_RSS=$(ps -o rss= -p $ML_PID | awk '{printf "%.0f", $1/1024}')
|
||||
ML_DISK=$(du -sm "$ML_LOG" | awk '{print $1}')
|
||||
|
||||
echo; echo "### mongo-lite kill -9 + reopen (replay)"
|
||||
kill -9 $ML_PID; wait $ML_PID 2>/dev/null
|
||||
ML_REOPEN=$(cd tests/e2e && node -e '
|
||||
const { spawn } = require("child_process");
|
||||
const { MongoClient } = require("mongodb");
|
||||
const t0 = Date.now();
|
||||
const p = spawn("/Users/shkmv/workspace/sandbox/mongo-lite/zig-out/bin/mongo-lite", ["--port","27019","--db","/tmp/mongo-cmp/ml.log","--compact-threshold","1g"], {stdio:"ignore"});
|
||||
const poll = async () => {
|
||||
const c = new MongoClient("mongodb://127.0.0.1:27019", {serverSelectionTimeoutMS: 800});
|
||||
try { await c.connect(); await c.db("admin").command({ping:1}); await c.close(); p.kill("SIGKILL"); console.log(((Date.now()-t0)/1000).toFixed(1)); }
|
||||
catch { try { await c.close(); } catch {}; setTimeout(poll, 200); }
|
||||
};
|
||||
setTimeout(poll, 300);
|
||||
')
|
||||
kill -9 $ML_PID 2>/dev/null
|
||||
|
||||
# ---- side by side ----------------------------------------------------------
|
||||
echo; echo "### side by side — ${SIZE} dataset, ~${DOC} docs"
|
||||
cat > "$CMPDIR/meta.json" <<EOF
|
||||
{"ml_rss_mb": "$ML_RSS", "md_rss_mb": "$MD_RSS", "ml_reopen": "${ML_REOPEN}s", "md_reopen": "${MD_REOPEN}s", "ml_disk_mb": "${ML_DISK}MB", "md_disk_mb": "${MD_DISK}MB"}
|
||||
EOF
|
||||
node -e '
|
||||
const fs = require("fs");
|
||||
const read = (p) => {
|
||||
const m = {};
|
||||
if (!fs.existsSync(p)) return m;
|
||||
for (const line of fs.readFileSync(p, "utf8").split("\n")) {
|
||||
const i = line.indexOf("\t");
|
||||
if (i > 0) m[line.slice(0, i)] = line.slice(i + 1).replace(/\t.*$/, "");
|
||||
}
|
||||
return m;
|
||||
};
|
||||
const a = read("/tmp/mongo-cmp/ml-report.txt");
|
||||
const b = read("/tmp/mongo-cmp/mongo-report.txt");
|
||||
const meta = JSON.parse(fs.readFileSync("/tmp/mongo-cmp/meta.json", "utf8"));
|
||||
const keys = ["insertOne (sequential) ×200","bulk insert throughput","docs loaded","createIndex({k: 1})",
|
||||
"countDocuments({})","findOne({_id: <ObjectId>})","findOne({k: 500}) (indexed)","find({p: {$gte,$lt}}).count() (scan)",
|
||||
"find({}).sort({_id:-1}).limit(20)","find({}, {proj}).limit(1000)","aggregate $group by k",
|
||||
"updateOne({_id}) ×50","updateMany({k: 7}, {$inc})","deleteOne({_id}) + insertOne","node client RSS"];
|
||||
const col = (v) => String(v).padEnd(22);
|
||||
console.log(`${"benchmark".padEnd(42)} ${col("mongo-lite")} ${col("mongodb")} ratio`);
|
||||
for (const k of keys) {
|
||||
const av = a[k] || "—", bv = b[k] || "—";
|
||||
const ar = parseFloat(av), br = parseFloat(bv);
|
||||
const ratio = isFinite(ar) && isFinite(br) && ar > 0 && br > 0 ? (ar / br).toFixed(1) + "x" : "";
|
||||
console.log(`${k.padEnd(42)} ${col(av)} ${col(bv)} ${ratio}`);
|
||||
}
|
||||
console.log(`${`server RSS`.padEnd(42)} ${col(meta.ml_rss_mb + " MB")} ${col(meta.md_rss_mb + " MB")}`);
|
||||
console.log(`${`kill -9 reopen`.padEnd(42)} ${col(meta.ml_reopen)} ${col(meta.md_reopen)}`);
|
||||
console.log(`${`db on disk`.padEnd(42)} ${col(meta.ml_disk_mb)} ${col(meta.md_disk_mb)}`);
|
||||
'
|
||||
|
||||
pkill -9 -f "mongo-lite --port $ML_PORT" 2>/dev/null
|
||||
echo; echo "done — reports: $CMPDIR/ml-report.txt, $CMPDIR/mongo-report.txt"
|
||||
177
tests/e2e/compare.js
Normal file
177
tests/e2e/compare.js
Normal file
@@ -0,0 +1,177 @@
|
||||
// Benchmark: the same workload through the official driver against mongo-lite
|
||||
// and a real MongoDB. Run once per server URL, then diff the reports.
|
||||
//
|
||||
// node tests/e2e/compare.js --url mongodb://127.0.0.1:27018 --label mongodb
|
||||
// node tests/e2e/compare.js --url mongodb://127.0.0.1:27019 --label mongo-lite
|
||||
//
|
||||
// Options:
|
||||
// --url <u> server URL (required)
|
||||
// --label <s> report label (default: url host:port)
|
||||
// --size <n> dataset size, k/m/g suffixes (default 1g)
|
||||
// --doc-size <n> bytes per document (default 16k)
|
||||
// --batch <n> docs per insertMany (default 500)
|
||||
// --index <field> field to index before the op benchmarks (default k)
|
||||
// --wc <j|none> write concern: 'j' = {w:1, j:true} durable ack on every
|
||||
// write (fair vs mongo-lite's fsync-per-write); 'none' =
|
||||
// driver default (default j)
|
||||
//
|
||||
// Every benchmark is awaited (no fire-and-forget), which is exactly how the
|
||||
// big.js harness measures mongo-lite, so the numbers are directly comparable.
|
||||
const { MongoClient, ObjectId } = require('mongodb');
|
||||
const fs = require('fs');
|
||||
const os = require('os');
|
||||
|
||||
function parseSize(s) {
|
||||
const m = /^(\d+(?:\.\d+)?)([kmgt]?)$/i.exec(String(s).trim());
|
||||
if (!m) throw new Error(`bad size '${s}'`);
|
||||
const mult = { '': 1, k: 1 << 10, m: 1 << 20, g: 1 << 30, t: 1 << 40 }[m[2].toLowerCase()];
|
||||
return Math.round(parseFloat(m[1]) * mult);
|
||||
}
|
||||
function fmt(n) { return n >= 1e9 ? (n / 1e9).toFixed(2) + ' GB' : n >= 1e6 ? (n / 1e6).toFixed(1) + ' MB' : n >= 1e3 ? (n / 1e3).toFixed(1) + ' KB' : n + ' B'; }
|
||||
|
||||
let opt = { url: null, label: null, size: '1g', docSize: '16k', batch: 500, index: 'k', wc: 'j' };
|
||||
for (let i = 2; i < process.argv.length; i++) {
|
||||
const a = process.argv[i];
|
||||
if (a === '--url') opt.url = process.argv[++i];
|
||||
else if (a === '--label') opt.label = process.argv[++i];
|
||||
else if (a === '--size') opt.size = process.argv[++i];
|
||||
else if (a === '--doc-size') opt.docSize = process.argv[++i];
|
||||
else if (a === '--batch') opt.batch = Number(process.argv[++i]);
|
||||
else if (a === '--index') opt.index = process.argv[++i];
|
||||
else if (a === '--wc') opt.wc = process.argv[++i];
|
||||
else { console.error(`unknown option ${a}`); process.exit(2); }
|
||||
}
|
||||
if (!opt.url) { console.error('--url required'); process.exit(2); }
|
||||
opt.label = opt.label || opt.url.replace(/^mongodb:\/\//, '');
|
||||
const SIZE = parseSize(opt.size);
|
||||
const DOC_SIZE = parseSize(opt.docSize);
|
||||
const WC = opt.wc === 'j' ? { writeConcern: { w: 1, j: true } } : {};
|
||||
|
||||
const report = [];
|
||||
const row = (k, v, note = '') => { report.push([k, v, note]); console.log(`${k}\t${v}${note ? '\t' + note : ''}`); };
|
||||
const fmtMs = (ms) => (ms < 1 ? ms.toFixed(2) + ' ms' : ms < 1000 ? ms.toFixed(1) + ' ms' : (ms / 1000).toFixed(2) + ' s');
|
||||
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
|
||||
|
||||
async function bench(label, fn, min = 0) {
|
||||
const a = process.hrtime.bigint();
|
||||
const out = await fn();
|
||||
const ms = Number(process.hrtime.bigint() - a) / 1e6;
|
||||
row(label, fmtMs(ms), out ?? '');
|
||||
return ms;
|
||||
}
|
||||
async function benchMany(label, n, fn) {
|
||||
const times = [];
|
||||
const a = process.hrtime.bigint();
|
||||
for (let i = 0; i < n; i++) await fn();
|
||||
const total = Number(process.hrtime.bigint() - a) / 1e6;
|
||||
const sorted = times.sort((x, y) => x - y);
|
||||
row(`${label} ×${n}`, fmtMs(total / n), `total ${fmtMs(total)}`);
|
||||
return total / n;
|
||||
}
|
||||
function rssMB() {
|
||||
try {
|
||||
return Math.round(process.memoryUsage().rss / 1048576);
|
||||
} catch { return 0; }
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const client = new MongoClient(opt.url, { serverSelectionTimeoutMS: 15000, maxPoolSize: 4 });
|
||||
await client.connect();
|
||||
const dbName = `cmp${Date.now() % 100000}`;
|
||||
const db = client.db(dbName);
|
||||
const coll = db.collection('items');
|
||||
const payload = 'y'.repeat(Math.max(1, DOC_SIZE - 140));
|
||||
const nDocs = Math.max(1, Math.ceil(SIZE / DOC_SIZE));
|
||||
|
||||
console.log(`== ${opt.label} == size ${fmt(SIZE)} · doc ~${fmt(DOC_SIZE)} · wc ${opt.wc}`);
|
||||
await db.command({ ping: 1 });
|
||||
row('driver handshake + ping', '');
|
||||
|
||||
// ---- single-doc insert latency (sequential, durable ack) ----------------
|
||||
console.log('\n-- single-doc writes --');
|
||||
await coll.deleteMany({});
|
||||
await benchMany('insertOne (sequential)', 200, () => coll.insertOne({ _id: new ObjectId(), k: 1, p: 2, ts: new Date(), payload: 'z'.repeat(256) }, WC));
|
||||
const one = await coll.findOne({});
|
||||
row('insertOne round-trip sanity', one ? 'ok' : 'MISSING');
|
||||
|
||||
// ---- bulk load ------------------------------------------------------------
|
||||
console.log('\n-- bulk load --');
|
||||
await coll.drop().catch(() => {});
|
||||
await coll.createIndex({ _id: 1 }, { unique: true }).catch(() => {});
|
||||
let t0 = Date.now();
|
||||
let inserted = 0;
|
||||
while (inserted < nDocs) {
|
||||
const n = Math.min(opt.batch, nDocs - inserted);
|
||||
const docs = new Array(n);
|
||||
for (let i = 0; i < n; i++) {
|
||||
const id = inserted + i + 1;
|
||||
docs[i] = { _id: new ObjectId(), k: id % 1000, p: id % 5000, ts: new Date(Date.UTC(2024, 0, 1) + id * 1000), payload };
|
||||
}
|
||||
await coll.insertMany(docs, Object.assign({ ordered: false }, WC));
|
||||
inserted += n;
|
||||
}
|
||||
const bulkMs = Date.now() - t0;
|
||||
row('docs loaded', inserted.toLocaleString());
|
||||
row('bytes loaded', fmt(inserted * DOC_SIZE));
|
||||
row('bulk insert throughput', `${(inserted * DOC_SIZE / 1e6 / (bulkMs / 1000)).toFixed(1)} MB/s`, `(${(inserted / (bulkMs / 1000)).toFixed(0)} docs/s, ${(bulkMs / 1000).toFixed(1)} s)`);
|
||||
let collInfo = null;
|
||||
try { collInfo = await db.command({ collStats: 'items' }); } catch {}
|
||||
if (collInfo) row('server-side data size', fmt(collInfo.size ?? 0), `storage ${fmt(collInfo.storageSize ?? 0)}`);
|
||||
|
||||
// ---- secondary index (mongo-lite: planner uses it; mongodb: normal) ------
|
||||
if (opt.index) {
|
||||
await bench(`createIndex({${opt.index}: 1})`, async () => { await coll.createIndex({ [opt.index]: 1 }); });
|
||||
}
|
||||
|
||||
// ---- read benchmarks --------------------------------------------------------
|
||||
console.log('\n-- reads on full dataset --');
|
||||
await bench('countDocuments({})', async () => `${(await coll.countDocuments({})).toLocaleString()} docs`);
|
||||
await bench('findOne({_id: <ObjectId>})', async () => {
|
||||
const d = await coll.findOne({ _id: new ObjectId(hexOf(nDocs / 2)) }, { projection: { _id: 1 } });
|
||||
return d ? 'hit' : 'MISSING';
|
||||
});
|
||||
if (opt.index) {
|
||||
await bench(`findOne({${opt.index}: 500}) (indexed)`, async () => {
|
||||
const d = await coll.findOne({ k: 500 }, { projection: { _id: 1 } });
|
||||
return d ? 'hit' : 'MISSING';
|
||||
});
|
||||
}
|
||||
await bench('find({p: {$gte,$lt}}).count() (scan)', async () => `${(await coll.countDocuments({ p: { $gte: 1000, $lt: 3000 } })).toLocaleString()} hits`);
|
||||
await bench('find({}).sort({_id:-1}).limit(20)', async () => (await coll.find({}).sort({ _id: -1 }).limit(20).toArray()).length + ' docs');
|
||||
await bench('find({}, {proj}).limit(1000)', async () => (await coll.find({}, { projection: { payload: 0 } }).limit(1000).toArray()).length + ' docs');
|
||||
await bench('aggregate $group by k', async () => {
|
||||
const g = await coll.aggregate([{ $group: { _id: '$k', n: { $sum: 1 } } }]).toArray();
|
||||
return g.length + ' groups';
|
||||
});
|
||||
|
||||
// ---- write benchmarks on the full dataset ---------------------------------
|
||||
console.log('\n-- writes on full dataset --');
|
||||
const midOid = new ObjectId(hexOf(Math.floor(nDocs / 2)));
|
||||
await benchMany('updateOne({_id})', 50, () => coll.updateOne({ _id: midOid }, { $set: { touch: 1 } }, WC));
|
||||
await bench('updateMany({k: 7}, {$inc})', async () => {
|
||||
const r = await coll.updateMany({ k: 7 }, { $inc: { hits: 1 } }, WC);
|
||||
return `${r.modifiedCount} modified`;
|
||||
});
|
||||
await bench('deleteOne({_id}) + insertOne', async () => {
|
||||
await coll.deleteOne({ _id: midOid }, WC);
|
||||
await coll.insertOne({ _id: new ObjectId(), k: 1, p: 2, payload }, WC);
|
||||
});
|
||||
|
||||
row('node client RSS', `${rssMB()} MB`);
|
||||
|
||||
await client.close();
|
||||
// Drop the db so a second run against the same server starts clean.
|
||||
await new MongoClient(opt.url, { serverSelectionTimeoutMS: 5000 }).connect().then(async (c) => {
|
||||
await c.db(dbName).dropDatabase();
|
||||
await c.close();
|
||||
}).catch(() => {});
|
||||
|
||||
console.log('\nCOMPARE_OK');
|
||||
}
|
||||
|
||||
// ObjectId from a deterministic 12-byte hex (so both servers see identical keys).
|
||||
function hexOf(n) {
|
||||
return n.toString(16).padStart(24, '0');
|
||||
}
|
||||
|
||||
main().catch((e) => { console.error('COMPARE_FAIL', e); process.exit(1); });
|
||||
426
tests/e2e/e2e6.js
Normal file
426
tests/e2e/e2e6.js
Normal file
@@ -0,0 +1,426 @@
|
||||
// E2E part 6: the full lifecycle, self-contained.
|
||||
//
|
||||
// Spawns its own mongo-lite server on a fresh log file and drives the whole
|
||||
// feature surface through the official driver: CRUD + query operators +
|
||||
// aggregation + error codes + secondary indexes + TTL expiry + admin
|
||||
// commands, then restarts the server twice — once gracefully, once with
|
||||
// kill -9 mid-write — and verifies that everything (data, indexes, TTL
|
||||
// state) survives both.
|
||||
//
|
||||
// Unlike the other e2e files it needs no server running beforehand:
|
||||
//
|
||||
// node tests/e2e/e2e6.js
|
||||
//
|
||||
// Env: E2E6_PORT listen port (default 27220)
|
||||
// ML_BIN server binary (default ../../zig-out/bin/mongo-lite)
|
||||
// E2E6_KEEP keep the log file after the run
|
||||
const { MongoClient, ObjectId } = require('mongodb');
|
||||
const { spawn } = require('child_process');
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
|
||||
const PORT = Number(process.env.E2E6_PORT || 27220);
|
||||
const BIN = process.env.ML_BIN || path.resolve(__dirname, '../../zig-out/bin/mongo-lite');
|
||||
const DBFILE = process.env.E2E6_DB || path.resolve(__dirname, '../../.zig-cache/e2e6-full.log');
|
||||
const URL = `mongodb://127.0.0.1:${PORT}`;
|
||||
|
||||
const results = [];
|
||||
function check(name, cond, detail = '') {
|
||||
results.push({ name, ok: !!cond, detail: String(detail) });
|
||||
if (!cond) console.error(` ✗ ${name} ${detail}`);
|
||||
}
|
||||
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
|
||||
async function waitFor(fn, timeoutMs = 20000, stepMs = 200) {
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (Date.now() < deadline) {
|
||||
if (await fn()) return true;
|
||||
await sleep(stepMs);
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
let server = null;
|
||||
let serverLog = '';
|
||||
let serverDead = false;
|
||||
|
||||
// Never leak the spawned server: kill it no matter how the test exits.
|
||||
function cleanup() {
|
||||
if (server && !serverDead) {
|
||||
try { server.kill('SIGKILL'); } catch {}
|
||||
}
|
||||
}
|
||||
process.on('exit', cleanup);
|
||||
process.on('SIGINT', () => { cleanup(); process.exit(130); });
|
||||
process.on('SIGTERM', () => { cleanup(); process.exit(143); });
|
||||
|
||||
function startServer(fresh = false) {
|
||||
return new Promise((resolve, reject) => {
|
||||
// Only the very first start must wipe the log; restarts must reuse it.
|
||||
if (fresh) fs.rmSync(DBFILE, { force: true });
|
||||
serverDead = false;
|
||||
server = spawn(BIN, ['--port', String(PORT), '--db', DBFILE, '--ttl-sweep-secs', '1'], {
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
});
|
||||
server.stdout.on('data', (d) => (serverLog += d));
|
||||
server.stderr.on('data', (d) => (serverLog += d));
|
||||
server.on('error', (e) => reject(new Error(`cannot start ${BIN}: ${e.message}`)));
|
||||
server.on('exit', (code, sig) => {
|
||||
// A child that dies (e.g. address already in use) must fail the start;
|
||||
// otherwise the ping poll below would talk to a *stale* server on the
|
||||
// same port and the whole run would go against the wrong database.
|
||||
serverDead = true;
|
||||
if (code !== null && sig === null) serverLog += `\n[child exited rc=${code}]`;
|
||||
});
|
||||
// Replay happens before the listener opens, so a successful connect is
|
||||
// also the reopen benchmark. Poll until the server answers ping.
|
||||
const deadline = Date.now() + 15000;
|
||||
(async () => {
|
||||
while (Date.now() < deadline) {
|
||||
if (serverDead) {
|
||||
reject(new Error(`server child exited during start (port ${PORT} busy?)\n${serverLog}`));
|
||||
return;
|
||||
}
|
||||
const c = new MongoClient(URL, { serverSelectionTimeoutMS: 1000 });
|
||||
try {
|
||||
await c.connect();
|
||||
await c.db('admin').command({ ping: 1 });
|
||||
await c.close();
|
||||
return resolve();
|
||||
} catch {
|
||||
try { await c.close(); } catch {}
|
||||
await sleep(100);
|
||||
}
|
||||
}
|
||||
reject(new Error(`server did not come up on :${PORT}\n${serverLog}`));
|
||||
})();
|
||||
});
|
||||
}
|
||||
|
||||
async function stopServer(sig = 'SIGTERM') {
|
||||
if (!server) return;
|
||||
const exited = new Promise((r) => server.once('exit', r));
|
||||
server.kill(sig);
|
||||
await Promise.race([exited, sleep(5000)]);
|
||||
serverDead = true;
|
||||
server = null;
|
||||
}
|
||||
|
||||
async function expectCode(fn, code, name) {
|
||||
let err = null;
|
||||
try {
|
||||
await fn();
|
||||
} catch (e) {
|
||||
err = e;
|
||||
}
|
||||
check(name, err && err.code === code, err ? `code ${err.code}: ${err.message}` : 'no error');
|
||||
}
|
||||
|
||||
async function phase1(client, db) {
|
||||
const users = db.collection('users');
|
||||
await users.drop().catch(() => {});
|
||||
|
||||
// ---- insert -----------------------------------------------------------
|
||||
await users.insertOne({ name: 'alice', age: 30, tags: ['a', 'b'], scores: [3, 6, 9], email: 'a@x.io' });
|
||||
const many = await users.insertMany([
|
||||
{ name: 'bob', age: 25, tags: ['b'], scores: [1], email: 'b@x.io' },
|
||||
{ name: 'carol', age: 35, tags: ['c', 'a'], scores: [8, 5], email: 'c@x.io' },
|
||||
{ name: 'dave', age: 40, tags: [], scores: [4, 4, 4], email: 'd@x.io' },
|
||||
]);
|
||||
check('insertMany acknowledged', many.acknowledged === true, many);
|
||||
check('auto _id assigned', ObjectId.isValid(many.insertedIds[0]));
|
||||
const aliceId = (await users.findOne({ name: 'alice' }))._id;
|
||||
check('explicit _id round-trips', (await users.findOne({ _id: many.insertedIds[0] })).name === 'bob');
|
||||
|
||||
// ---- find: operators ---------------------------------------------------
|
||||
check('$gt + sort desc', (await users.find({ age: { $gt: 28 } }).sort({ age: -1 }).toArray()).map((d) => d.name).join(',') === 'dave,carol,alice');
|
||||
check('$gte', (await users.countDocuments({ age: { $gte: 30 } })) === 3);
|
||||
check('$lt', (await users.countDocuments({ age: { $lt: 30 } })) === 1);
|
||||
check('$lte', (await users.countDocuments({ age: { $lte: 30 } })) === 2);
|
||||
check('$ne', (await users.countDocuments({ age: { $ne: 30 } })) === 3);
|
||||
check('$in', (await users.find({ name: { $in: ['alice', 'bob'] } }).count()) === 2);
|
||||
check('$nin', (await users.countDocuments({ name: { $nin: ['alice', 'bob', 'carol', 'dave'] } })) === 0);
|
||||
check('$exists true', (await users.countDocuments({ tags: { $exists: true } })) === 4);
|
||||
check('$exists false', (await users.countDocuments({ ghost: { $exists: false } })) === 4);
|
||||
check('$regex anchors', (await users.find({ name: /^[bc]/ }).toArray()).length === 2);
|
||||
check('$regex case-insensitive', (await users.countDocuments({ name: /^ALICE$/i })) === 1);
|
||||
check('$not', (await users.countDocuments({ age: { $not: { $gt: 30 } } })) === 2);
|
||||
check('$and', (await users.countDocuments({ $and: [{ age: { $gte: 25 } }, { age: { $lt: 40 } }] })) === 3);
|
||||
check('$or', (await users.countDocuments({ $or: [{ name: 'alice' }, { name: 'dave' }] })) === 2);
|
||||
check('$nor', (await users.countDocuments({ $nor: [{ name: 'alice' }, { name: 'dave' }] })) === 2);
|
||||
check('$size', (await users.countDocuments({ tags: { $size: 2 } })) === 2);
|
||||
check('$all', (await users.countDocuments({ tags: { $all: ['a', 'b'] } })) === 1);
|
||||
check('$elemMatch', (await users.countDocuments({ scores: { $elemMatch: { $gte: 5, $lt: 9 } } })) === 2);
|
||||
check('dot path', (await users.findOne({ 'tags.0': 'c' })).name === 'carol');
|
||||
check('dot path numeric index', (await users.findOne({ 'scores.0': 3 })).name === 'alice');
|
||||
check('array equality', (await users.countDocuments({ scores: [4, 4, 4] })) === 1);
|
||||
|
||||
// ---- find: sort / skip / limit / projection ----------------------------
|
||||
check('skip+limit+sort', (await users.find({}).sort({ age: 1 }).skip(1).limit(2).toArray()).map((d) => d.name).join(',') === 'alice,carol');
|
||||
const proj = await users.findOne({ name: 'alice' }, { projection: { _id: 0, name: 1 } });
|
||||
check('projection', proj.name === 'alice' && proj.age === undefined, JSON.stringify(proj));
|
||||
|
||||
// ---- counts ------------------------------------------------------------
|
||||
check('countDocuments', (await users.countDocuments({})) === 4);
|
||||
check('estimatedDocumentCount', (await users.estimatedDocumentCount()) === 4);
|
||||
|
||||
// ---- update ------------------------------------------------------------
|
||||
const u1 = await users.updateOne({ name: 'alice' }, { $set: { vip: true }, $inc: { age: 1 } });
|
||||
check('updateOne nModified', u1.modifiedCount === 1, u1);
|
||||
const alice = await users.findOne({ _id: aliceId });
|
||||
check('$set+$inc applied', alice.vip === true && alice.age === 31, JSON.stringify(alice));
|
||||
await users.updateOne({ name: 'carol' }, { $unset: { tags: '' } });
|
||||
check('$unset', (await users.findOne({ name: 'carol' })).tags === undefined);
|
||||
await users.updateOne({ name: 'dave' }, { $push: { tags: 'x' }, $rename: { vip: 'member' } });
|
||||
const dave = await users.findOne({ name: 'dave' });
|
||||
check('$push+$rename', dave.tags.length === 1 && dave.member === undefined, JSON.stringify(dave));
|
||||
await users.updateOne({ name: 'dave' }, { $pull: { tags: 'x' } });
|
||||
check('$pull', (await users.findOne({ name: 'dave' })).tags.length === 0);
|
||||
const um = await users.updateMany({}, { $set: { seen: true } });
|
||||
check('updateMany', um.modifiedCount === 4, um);
|
||||
const ups = await users.updateOne({ name: 'erin' }, { $set: { age: 28 } }, { upsert: true });
|
||||
check('upsert create', ups.upsertedCount === 1 && ups.matchedCount === 0, ups);
|
||||
const ups2 = await users.updateOne({ name: 'erin' }, { $set: { age: 29 } }, { upsert: true });
|
||||
check('upsert match', ups2.upsertedCount === 0 && ups2.modifiedCount === 1, ups2);
|
||||
|
||||
// ---- findOneAndUpdate / Delete ------------------------------------------
|
||||
const fam = await users.findOneAndUpdate({ name: 'bob' }, { $set: { lucky: true } }, { returnDocument: 'after' });
|
||||
check('findOneAndUpdate returns new', (fam.value ?? fam).lucky === true);
|
||||
const famDel = await users.findOneAndDelete({ name: 'erin' });
|
||||
check('findOneAndDelete', (famDel.value ?? famDel).name === 'erin');
|
||||
|
||||
// ---- aggregate -----------------------------------------------------------
|
||||
const grp = await users
|
||||
.aggregate([
|
||||
{ $match: { age: { $gte: 25 } } },
|
||||
{ $group: { _id: '$tags.length', total: { $sum: '$age' } } },
|
||||
{ $sort: { _id: 1 } },
|
||||
])
|
||||
.toArray();
|
||||
check('aggregate $match+$group+$sum', grp.length >= 1 && grp.some((g) => g.total > 0), JSON.stringify(grp));
|
||||
check('aggregate $count', (await users.aggregate([{ $match: {} }, { $count: 'n' }]).toArray())[0].n === 4);
|
||||
// $project here is field selection (inclusion/exclusion); computed fields
|
||||
// like `who: '$name'` are outside the documented surface.
|
||||
const projAgg = await users.aggregate([{ $match: { name: 'alice' } }, { $project: { _id: 0, name: 1 } }]).toArray();
|
||||
check('aggregate $project', projAgg.length === 1 && projAgg[0].name === 'alice' && projAgg[0].age === undefined, JSON.stringify(projAgg));
|
||||
|
||||
// ---- duplicate key -------------------------------------------------------
|
||||
await expectCode(() => users.insertOne({ _id: many.insertedIds[0], name: 'clobber' }), 11000, 'duplicate _id rejected (11000)');
|
||||
|
||||
// ---- indexes -------------------------------------------------------------
|
||||
await users.createIndex({ email: 1 }, { unique: true });
|
||||
await users.createIndex({ name: 1, age: 1 }, { name: 'name_1_age_1' });
|
||||
await users.createIndex({ nickname: 1 }, { sparse: true, name: 'nickname_1_sparse' });
|
||||
await users.createIndex({ score: -1 });
|
||||
const idxs = await users.indexes();
|
||||
const names = idxs.map((i) => i.name).sort();
|
||||
check('indexes listed', names.join(',') === '_id_,email_1,name_1_age_1,nickname_1_sparse,score_-1', names.join(','));
|
||||
await users.updateOne({ name: 'dave' }, { $set: { email: 'dave@x.io', nickname: 'davie' } });
|
||||
check('unique index find', (await users.find({ email: 'dave@x.io' }).toArray()).length === 1);
|
||||
await expectCode(() => users.insertOne({ name: 'dup', email: 'dave@x.io' }), 11000, 'unique index rejects dup (11000)');
|
||||
check('compound prefix find', (await users.find({ name: 'alice', age: 31 }).toArray()).length === 1);
|
||||
check('descending index created', idxs.some((i) => i.key && i.key.score === -1));
|
||||
await users.dropIndex('name_1_age_1');
|
||||
check('dropIndex', !(await users.indexes()).some((i) => i.name === 'name_1_age_1'));
|
||||
await users.dropIndexes();
|
||||
const afterAll = await users.indexes();
|
||||
check('dropIndexes keeps only _id_', afterAll.length === 1 && afterAll[0].name === '_id_', JSON.stringify(afterAll.map((i) => i.name)));
|
||||
// Recreate the unique index: the restart phase depends on it surviving.
|
||||
// Every doc has a distinct email (inserted above), so this is legal — a
|
||||
// unique index over docs that *lack* the field would be E11000 (duplicate
|
||||
// null), exactly as in MongoDB.
|
||||
await users.createIndex({ email: 1 }, { unique: true });
|
||||
|
||||
// ---- TTL -----------------------------------------------------------------
|
||||
const sessions = db.collection('sessions');
|
||||
await sessions.drop().catch(() => {});
|
||||
await sessions.createIndex({ expireAt: 1 }, { expireAfterSeconds: 1 });
|
||||
const now = Date.now();
|
||||
await sessions.insertMany([
|
||||
{ _id: 'past', expireAt: new Date(now - 60_000) },
|
||||
{ _id: 'future', expireAt: new Date(now + 3_600_000) },
|
||||
]);
|
||||
check('TTL index listed', (await sessions.indexes()).some((i) => i.name === 'expireAt_1' && Number(i.expireAfterSeconds) === 1));
|
||||
check('TTL past-doc expired', await waitFor(async () => (await sessions.countDocuments({ _id: 'past' })) === 0));
|
||||
check('TTL future-doc survives', (await sessions.countDocuments({ _id: 'future' })) === 1);
|
||||
|
||||
// ---- admin ----------------------------------------------------------------
|
||||
const colls = await db.listCollections({}, { nameOnly: true }).toArray();
|
||||
check('listCollections', colls.some((c) => c.name === 'users') && colls.some((c) => c.name === 'sessions'));
|
||||
const dbs = await db.admin().listDatabases();
|
||||
check('listDatabases has e2e6', dbs.databases.some((d) => d.name === 'e2e6'));
|
||||
// dropDatabase must run against a scratch db so it can't nuke the data the
|
||||
// later phases depend on.
|
||||
const scratch = client.db('e2e6_scratch');
|
||||
await scratch.collection('scratchme').insertOne({ x: 1 });
|
||||
await scratch.dropDatabase();
|
||||
const dbs2 = await db.admin().listDatabases();
|
||||
check('dropDatabase removes it', !dbs2.databases.some((d) => d.name === 'e2e6_scratch'), JSON.stringify(dbs2.databases.map((d) => d.name)));
|
||||
|
||||
return { users, sessions, aliceId };
|
||||
}
|
||||
|
||||
async function phase2(client, { users, aliceId }) {
|
||||
// Compaction only means something when writes *discard* data: the log is
|
||||
// append-only, so replace/delete records pile up as junk until the
|
||||
// 16 MiB threshold triggers a rewrite of just the live documents.
|
||||
//
|
||||
// insert 2000 x 12KB (~24 MB)
|
||||
// replace all 2000 (+24 MB junk)
|
||||
// delete half (+12 MB junk)
|
||||
//
|
||||
// ~60 MB written; a working compactor leaves the file near the live
|
||||
// size (~12 MB + one 16 MB epoch), a broken one leaves ~60 MB.
|
||||
const bulk = client.db('e2e6').collection('bulk');
|
||||
await bulk.drop().catch(() => {});
|
||||
const payload = 'z'.repeat(12 * 1024);
|
||||
for (let b = 0; b < 4; b++) {
|
||||
const docs = Array.from({ length: 500 }, (_, i) => ({ _id: b * 500 + i, g: (b * 500 + i) % 2, payload }));
|
||||
await bulk.insertMany(docs);
|
||||
}
|
||||
if (process.env.E2E6_DEBUG) {
|
||||
const dbg = await users.findOne({ name: 'alice' });
|
||||
console.log('DEBUG phase2 after bulk-insert alice _id:', String(dbg._id), 'same:', String(dbg._id) === String(aliceId));
|
||||
}
|
||||
// Watch the log file while the replace junk accumulates. Compaction is
|
||||
// fast (one fsync for the whole rewrite), so a sampled drop can be tiny;
|
||||
// the deterministic signals are that the file *peaked* well above where
|
||||
// it ended (junk accumulated) and *ended* near the live size (compaction
|
||||
// reclaimed it). ~48 MB of records are written here and ~12 MB of it
|
||||
// survives, so without compaction the file would end near 48 MB.
|
||||
//
|
||||
// Both bounds are relative to the live size on purpose: compaction now
|
||||
// triggers on the share of the log that is garbage rather than on bytes
|
||||
// appended, so the absolute peak depends on when that share crosses the
|
||||
// threshold and is not a stable number to assert on.
|
||||
let peakSize = fs.statSync(DBFILE).size;
|
||||
const watcher = setInterval(() => {
|
||||
const s = fs.statSync(DBFILE).size;
|
||||
if (s > peakSize) peakSize = s;
|
||||
}, 10);
|
||||
const payload2 = 'q'.repeat(12 * 1024);
|
||||
const replaced = await bulk.updateMany({}, { $set: { payload: payload2 } });
|
||||
check('bulk replace logged per doc', replaced.modifiedCount === 2000, replaced);
|
||||
const deleted = await bulk.deleteMany({ g: 0 });
|
||||
check('bulk delete half', deleted.deletedCount === 1000, deleted);
|
||||
clearInterval(watcher);
|
||||
|
||||
const logSize = fs.statSync(DBFILE).size;
|
||||
if (process.env.E2E6_DEBUG) {
|
||||
const dbg = await users.findOne({ name: 'alice' });
|
||||
console.log('DEBUG phase2 after replace+delete alice _id:', String(dbg._id), 'same:', String(dbg._id) === String(aliceId));
|
||||
console.log('DEBUG phase2 log size:', logSize);
|
||||
}
|
||||
check(
|
||||
'compaction reclaimed junk (file ~ live size)',
|
||||
logSize < 24 * 1024 * 1024 && peakSize > logSize * 1.4,
|
||||
`file peaked at ${(peakSize / 1e6).toFixed(1)}MB, ended at ${(logSize / 1e6).toFixed(1)}MB after ~48MB of records were written (~12MB live)`,
|
||||
);
|
||||
check('bulk survivors intact', (await bulk.findOne({ _id: 1999 })).payload === payload2);
|
||||
check('bulk count after delete', (await bulk.countDocuments({})) === 1000);
|
||||
|
||||
// Phase-1 data survives in memory (erin was deleted in phase 1, so 4).
|
||||
check('users count after bulk', (await users.countDocuments({})) === 4);
|
||||
|
||||
// ---- graceful restart -----------------------------------------------------
|
||||
await stopServer('SIGTERM');
|
||||
await startServer();
|
||||
const client2 = new MongoClient(URL, { serverSelectionTimeoutMS: 5000 });
|
||||
await client2.connect();
|
||||
const db2 = client2.db('e2e6');
|
||||
const users2 = db2.collection('users');
|
||||
|
||||
check('after restart: users count', (await users2.countDocuments({})) === 4);
|
||||
if (process.env.E2E6_DEBUG) {
|
||||
console.log('DEBUG aliceId:', String(aliceId), 'isOID', aliceId instanceof ObjectId);
|
||||
console.log('DEBUG all users:', JSON.stringify(await users2.find({}).toArray()));
|
||||
console.log('DEBUG bulk:', JSON.stringify(await db2.collection('bulk').find({}, { projection: { payload: 0 } }).limit(5).toArray()));
|
||||
console.log('DEBUG sessions:', JSON.stringify(await db2.collection('sessions').find({}).toArray()));
|
||||
console.log('DEBUG colls:', JSON.stringify(await db2.listCollections({}, { nameOnly: true }).toArray()));
|
||||
console.log('DEBUG dbs:', JSON.stringify((await db2.admin().listDatabases()).databases.map((d) => d.name)));
|
||||
console.log('DEBUG log file size:', fs.statSync(DBFILE).size);
|
||||
}
|
||||
const alice2 = await users2.findOne({ _id: aliceId });
|
||||
check('after restart: doc content', alice2.name === 'alice' && alice2.age === 31 && alice2.seen === true, JSON.stringify(alice2));
|
||||
check('after restart: unique index find', (await users2.find({ email: 'dave@x.io' }).toArray()).length === 1);
|
||||
check('after restart: unique index still enforced', await (async () => {
|
||||
try {
|
||||
await users2.insertOne({ name: 'dup2', email: 'dave@x.io' });
|
||||
return false;
|
||||
} catch (e) {
|
||||
return e.code === 11000;
|
||||
}
|
||||
})());
|
||||
check('after restart: bulk count', (await db2.collection('bulk').countDocuments({})) === 1000);
|
||||
check('after restart: TTL index listed', (await db2.collection('sessions').indexes()).some((i) => i.name === 'expireAt_1'));
|
||||
return client2;
|
||||
}
|
||||
|
||||
async function phase3(client) {
|
||||
// ---- kill -9 mid-write ----------------------------------------------------
|
||||
// Every write is logged + fsynced before it becomes visible, so whatever
|
||||
// count we see before the kill must be there after the restart.
|
||||
const crash = client.db('e2e6').collection('crash');
|
||||
await crash.drop().catch(() => {});
|
||||
let committed = 0;
|
||||
for (let i = 1; i <= 150; i++) {
|
||||
await crash.insertOne({ _id: i, seq: i });
|
||||
committed = i;
|
||||
}
|
||||
check('crash: committed before kill', committed === 150);
|
||||
await client.close();
|
||||
|
||||
await stopServer('SIGKILL');
|
||||
await startServer();
|
||||
const c3 = new MongoClient(URL, { serverSelectionTimeoutMS: 5000 });
|
||||
await c3.connect();
|
||||
const db3 = c3.db('e2e6');
|
||||
|
||||
const n = await db3.collection('crash').countDocuments({});
|
||||
check('crash recovery: all committed docs survived kill -9', n === 150, n);
|
||||
check('crash recovery: last doc intact', (await db3.collection('crash').findOne({ _id: 150 })).seq === 150);
|
||||
check('crash recovery: pre-crash data intact', (await db3.collection('users').countDocuments({})) === 4);
|
||||
await db3.collection('crash').insertOne({ _id: 151, seq: 151 });
|
||||
check('crash recovery: writes continue', (await db3.collection('crash').countDocuments({})) === 151);
|
||||
await c3.close();
|
||||
}
|
||||
|
||||
async function main() {
|
||||
if (!fs.existsSync(BIN)) {
|
||||
console.error(`server binary not found at ${BIN} — run \`zig build\` first`);
|
||||
process.exit(1);
|
||||
}
|
||||
await startServer(true);
|
||||
const client = new MongoClient(URL, { serverSelectionTimeoutMS: 5000 });
|
||||
await client.connect();
|
||||
const db = client.db('e2e6');
|
||||
|
||||
console.log('phase 1: feature surface');
|
||||
const state = await phase1(client, db);
|
||||
console.log('phase 2: compaction + graceful restart');
|
||||
const client2 = await phase2(client, state);
|
||||
// The phase-1 client's connection died with the restart; close it so the
|
||||
// process can exit (and so the exit handler can reap the server child).
|
||||
await client.close().catch(() => {});
|
||||
console.log('phase 3: kill -9 crash recovery');
|
||||
await phase3(client2);
|
||||
|
||||
if (process.env.E2E6_KEEP !== '1') fs.rmSync(DBFILE, { force: true });
|
||||
await stopServer('SIGTERM');
|
||||
|
||||
const failed = results.filter((r) => !r.ok);
|
||||
console.log(`\n${results.length - failed.length}/${results.length} checks passed`);
|
||||
if (failed.length) {
|
||||
console.log('FAILED:', failed.map((f) => f.name).join(', '));
|
||||
console.log('--- server log tail ---');
|
||||
console.log(serverLog.split('\n').slice(-30).join('\n'));
|
||||
process.exit(1);
|
||||
}
|
||||
console.log('E2E6_OK');
|
||||
}
|
||||
|
||||
main().catch((e) => {
|
||||
console.error('E2E6_FAIL', e);
|
||||
console.log('--- server log tail ---');
|
||||
console.log(serverLog.split('\n').slice(-40).join('\n'));
|
||||
process.exit(1);
|
||||
});
|
||||
Reference in New Issue
Block a user