results: the M0 gates, measured
PLAN D7's six items, with the numbers and the command that reproduces each in
tests/e2e/results/m0-gates.txt. Unit tests green in both optimize modes, the
whole e2e matrix green, the spec scorecard byte-identical at 131/161/195, and
the large smoke run at the scale D7.3 asked for:
21.47 GB collection (1,310,720 x 16 KiB)
data file 21.75 GB (+1.3% over the documents)
log after the load 2.5 MB (checkpoints reclaim it)
kill -9 then reopen 0.5 s (0.5 s at 4 GB too -- flat)
RSS after reopen 237 MB (1.1% of the data)
count after restart 1,310,720 last document byte-intact
acked writes after kill 200/200
That is the milestone's claim, measured: an open costs the working set rather
than the size of the database. Before M0 the same measurement was 523 MB
resident for a 512 MB database, because recovering each document's `_id` meant
reading every document at open.
Two gates need reading rather than a tick, and m0-gates.txt says so where a
reader would otherwise take a tick for granted.
The churn gate settles at 1.65x live data (delete-heavy) to 2.47x
(update-heavy), flat, above the ~1.3x amendment A2 hoped for. Rebuild-only
reclamation cannot reach that: it needs a whole second copy of the live data
before the first can be freed. The gate existed to decide whether doc-level
free lists are needed after M0, and that is the answer.
Benchmark parity holds for every read and latency row inside the run-to-run
spread, and bulk insert regresses 24% (732 -> 555 MB/s), reproducibly across
three runs. Risk 1 as written: document bytes now reach the disk uncompressed
on top of the LZ4 log. createIndex improves 62% from the same change.
Three measurement bugs fixed while running the gates, because each would have
put a false number in the README:
- `compare-run.sh` measured "db on disk" as `du` of the log alone against
`du` of mongod's whole dbpath. It reported 20 MB for a 1 GB collection --
the documents had moved to <db>.data. Honest figure, measured: 914 MB of
allocated blocks against mongod's compressed 85 MB.
- `big.js` counted "compaction events" as "the log shrank", which is a
*checkpoint* now. It claimed 12 compaction rewrites during a pure insert
load, which has no garbage to compact.
- `big.js` labelled peak RSS "in-memory engine: docs live in RAM" and its
summary said the collection was held "fully in RAM". Both were true of the
engine this milestone replaced.
README: the storage section described an all-in-RAM engine; the comparison
table mixed one old run's body with three new rows; and `findOne({_id})` was
documented as a full scan for integer ids, which the ordered `_id_` index made
false (2 ms against 55 s for a scan of the same 21.5 GB collection). The table
is now best-of-three for both servers, with the measured variance stated, since
two runs of the same binary moved the sub-10 ms rows by 27-51%.
This commit is contained in:
@@ -119,10 +119,13 @@ async function stopServer(sig = 'SIGKILL') {
|
||||
serverDead = true;
|
||||
server = null;
|
||||
}
|
||||
async function rssMB() {
|
||||
// Synchronous on purpose. The async version sampled inside setInterval and a
|
||||
// short run could finish before any sample landed, reporting RSS 0 -- which reads
|
||||
// as "measured and tiny" rather than "not measured".
|
||||
function rssMB() {
|
||||
if (!server) return 0;
|
||||
try {
|
||||
const out = (await new Promise((r) => require('child_process').exec(`ps -o rss= -p ${server.pid}`, (e, so) => r(so || '')))).trim();
|
||||
const out = require('child_process').execSync(`ps -o rss= -p ${server.pid}`, { encoding: 'utf8' }).trim();
|
||||
return Math.round(Number(out) / 1024);
|
||||
} catch { return 0; }
|
||||
}
|
||||
@@ -164,9 +167,9 @@ async function main() {
|
||||
const payload = 'x'.repeat(payloadLen);
|
||||
const { ObjectId } = require('mongodb');
|
||||
const t0 = Date.now();
|
||||
const logSamples = [{ t: 0, size: fs.existsSync(DBFILE) ? fs.statSync(DBFILE).size : 0, rss: 0 }];
|
||||
const sampler = setInterval(async () => {
|
||||
logSamples.push({ t: Date.now() - t0, size: fs.existsSync(DBFILE) ? fs.statSync(DBFILE).size : 0, rss: await rssMB() });
|
||||
const logSamples = [{ t: 0, size: fs.existsSync(DBFILE) ? fs.statSync(DBFILE).size : 0, rss: rssMB() }];
|
||||
const sampler = setInterval(() => {
|
||||
logSamples.push({ t: Date.now() - t0, size: fs.existsSync(DBFILE) ? fs.statSync(DBFILE).size : 0, rss: rssMB() });
|
||||
}, 2000);
|
||||
|
||||
// Rate curve: avg MB/s per progress chunk, to show how throughput changes
|
||||
@@ -215,12 +218,18 @@ async function main() {
|
||||
row('throughput', `${(bytes / 1e6 / (insertMs / 1000)).toFixed(1)} MB/s (${(inserted / (insertMs / 1000)).toFixed(0)} docs/s)`);
|
||||
row('rate curve (MB/s per chunk)', rates.join(' → ') || 'n/a');
|
||||
|
||||
const compactions = logSamples.filter((s, i) => i > 0 && s.size < logSamples[i - 1].size - 2 * 1024 * 1024).length;
|
||||
// A shrinking log means a *checkpoint* now, not a compaction: the checkpoint
|
||||
// publishes the data file and truncates the log to its header. Compaction
|
||||
// (which rewrites the data file) leaves no signature in the log's size, so
|
||||
// this counter cannot see it and must not claim to.
|
||||
const truncations = logSamples.filter((s, i) => i > 0 && s.size < logSamples[i - 1].size - 2 * 1024 * 1024).length;
|
||||
const sizes = logSamples.map((s) => s.size);
|
||||
row('log file size (min → final)', `${fmt(Math.min(...sizes))} → ${fmt(sizes[sizes.length - 1])}`);
|
||||
row('compaction events observed', compactions, '(log shrank by >2MB between samples)');
|
||||
const dataFileSize = fs.existsSync(DBFILE + '.data') ? fs.statSync(DBFILE + '.data').size : 0;
|
||||
row('data file size', fmt(dataFileSize));
|
||||
row('log truncations (checkpoints)', truncations, '(log shrank by >2MB between samples)');
|
||||
const peakRss = Math.max(...logSamples.map((s) => s.rss));
|
||||
row('peak server RSS', `${peakRss} MB`, '(in-memory engine: docs live in RAM)');
|
||||
row('peak server RSS', `${peakRss} MB`, '(a write touches its pages; see the after-reopen row)');
|
||||
|
||||
// ---- find / read -------------------------------------------------------
|
||||
console.log('\n== find / read on full dataset ==');
|
||||
@@ -298,7 +307,12 @@ async function main() {
|
||||
await client.close();
|
||||
await stopServer('SIGKILL');
|
||||
const reopenMs = await startServer();
|
||||
row('kill -9 then reopen (replay of full log)', `${(reopenMs / 1000).toFixed(1)} s`);
|
||||
row('kill -9 then reopen', `${(reopenMs / 1000).toFixed(1)} s`);
|
||||
// The point of the mmap work: what an open pays for is the working set, not
|
||||
// the size of the database. Sampled right after the server answers its first
|
||||
// ping, before any query has touched a document.
|
||||
const rssAfterReopen = rssMB();
|
||||
row('RSS after reopen (working set)', `${rssAfterReopen} MB`);
|
||||
const c2 = new MongoClient(URL, { serverSelectionTimeoutMS: 10000 });
|
||||
await c2.connect();
|
||||
const db2 = c2.db('big');
|
||||
@@ -335,10 +349,13 @@ async function main() {
|
||||
await stopServer('SIGKILL');
|
||||
|
||||
console.log('\n== summary ==');
|
||||
console.log(` multiforadb handles a ${fmt(bytes)} collection fully in RAM (RSS ${peakRss} MB)`);
|
||||
// Values captured while the server was alive: by the time the summary prints,
|
||||
// it has been stopped and its files removed, so sampling here reports zero.
|
||||
console.log(` multiforadb handles a ${fmt(bytes)} collection in a ${fmt(dataFileSize)} data file`);
|
||||
console.log(` peak RSS during load ${peakRss} MB; after reopen ${rssAfterReopen} MB — the working set, not the data size`);
|
||||
console.log(` insert: ${(bytes / 1e6 / (insertMs / 1000)).toFixed(1)} MB/s — fsync per write is by design (crash safety)`);
|
||||
if (compactions > 0) {
|
||||
console.log(` ${compactions} compaction rewrites observed: every 16MB of writes rewrites the whole log — for multi-GB loads the cumulative rewrite traffic dominates`);
|
||||
if (truncations > 0) {
|
||||
console.log(` ${truncations} checkpoints reclaimed the log during the load, which is why it ends at ${fmt(sizes[sizes.length - 1])} rather than ${fmt(bytes)}`);
|
||||
}
|
||||
console.log('BIG_OK');
|
||||
}
|
||||
|
||||
@@ -71,7 +71,12 @@ wait_ready "mongodb://127.0.0.1:$MFDB_PORT" || { echo "multiforadb never became
|
||||
node tests/e2e/compare.js --url "mongodb://127.0.0.1:$MFDB_PORT" --label multiforadb --size "$SIZE" --doc-size "$DOC" \
|
||||
> "$CMPDIR/mfdb-report.txt" 2>&1 || { echo "multiforadb bench failed:"; tail -5 "$CMPDIR/mfdb-report.txt"; }
|
||||
MFDB_RSS=$(ps -o rss= -p $MFDB_PID | awk '{printf "%.0f", $1/1024}')
|
||||
MFDB_DISK=$(du -sm "$MFDB_LOG" | awk '{print $1}')
|
||||
# Both files. The documents live in "$MFDB_LOG".data since the mmap foundation
|
||||
# landed, and the log is truncated at every checkpoint -- so measuring the log
|
||||
# alone reported 20 MB for a 1 GB collection, against a `du` over mongod's whole
|
||||
# dbpath. `du` rather than `ls`: the data file is grown with setLength and is
|
||||
# sparse until written, and allocated blocks are what actually costs disk.
|
||||
MFDB_DISK=$(du -scm "$MFDB_LOG" "$MFDB_LOG.data" 2>/dev/null | tail -1 | awk '{print $1}')
|
||||
|
||||
echo; echo "### multiforadb kill -9 + reopen (replay)"
|
||||
kill -9 $MFDB_PID; wait $MFDB_PID 2>/dev/null
|
||||
|
||||
36
tests/e2e/results/bench-20260803-223947.txt
Normal file
36
tests/e2e/results/bench-20260803-223947.txt
Normal file
@@ -0,0 +1,36 @@
|
||||
# multiforadb vs MongoDB benchmark
|
||||
# date: 2026-08-03T19:42:00Z git: 5228ed7+dirty
|
||||
# args: size=1g doc-size=16k wc=j clients=1 4 8 16 32 per-client=2000
|
||||
# reproduce: bash tests/e2e/bench-run.sh 1g 16k "1 4 8 16 32"
|
||||
|
||||
[main]
|
||||
insertOne (sequential) ×200 0.20 ms 4.9 ms
|
||||
bulk insert throughput 463.4 MB/s 581.3 MB/s
|
||||
docs loaded 65,536 65,536
|
||||
createIndex({k: 1}) 27.9 ms 78.1 ms
|
||||
countDocuments({}) 2.6 ms 12.8 ms
|
||||
findOne({_id: <ObjectId>}) 0.64 ms 0.72 ms
|
||||
findOne({k: 500}) (indexed) 0.59 ms 1.6 ms
|
||||
find({p: {$gte,$lt}}).count() (scan) 14.1 ms 12.5 ms
|
||||
find({}).sort({_id:-1}).limit(20) 1.5 ms 2.4 ms
|
||||
find({}, {proj}).limit(1000) 3.6 ms 4.7 ms
|
||||
aggregate $group by k 8.4 ms 13.4 ms
|
||||
updateOne({_id}) ×50 0.15 ms 0.23 ms
|
||||
updateMany({k: 7}, {$inc}) 1.9 ms 7.0 ms
|
||||
deleteOne({_id}) + insertOne 0.63 ms 5.8 ms
|
||||
node client RSS 153 MB 155 MB
|
||||
|
||||
[concurrency]
|
||||
clients 1 8354 204 41.0x
|
||||
clients 4 465 0.0x
|
||||
clients 8 978 0.0x
|
||||
clients 16 1877 0.0x
|
||||
clients 32 3765 0.0x
|
||||
|
||||
[meta]
|
||||
mfdb_rss_mb 1059
|
||||
md_rss_mb 1203
|
||||
mfdb_reopen 0.3s
|
||||
md_reopen 1.3s
|
||||
mfdb_disk_mb 20MB
|
||||
md_disk_mb 83MB
|
||||
36
tests/e2e/results/bench-20260803-225128.txt
Normal file
36
tests/e2e/results/bench-20260803-225128.txt
Normal file
@@ -0,0 +1,36 @@
|
||||
# multiforadb vs MongoDB benchmark
|
||||
# date: 2026-08-03T19:53:01Z git: 4b70ce6+dirty
|
||||
# args: size=1g doc-size=16k wc=j clients=1 4 8 16 32 per-client=2000
|
||||
# reproduce: bash tests/e2e/bench-run.sh 1g 16k "1 4 8 16 32"
|
||||
|
||||
[main]
|
||||
insertOne (sequential) ×200 0.22 ms 4.4 ms
|
||||
bulk insert throughput 555.5 MB/s 738.5 MB/s
|
||||
docs loaded 65,536 65,536
|
||||
createIndex({k: 1}) 28.0 ms 70.0 ms
|
||||
countDocuments({}) 2.8 ms 12.3 ms
|
||||
findOne({_id: <ObjectId>}) 0.60 ms 0.61 ms
|
||||
findOne({k: 500}) (indexed) 0.54 ms 1.1 ms
|
||||
find({p: {$gte,$lt}}).count() (scan) 11.2 ms 13.2 ms
|
||||
find({}).sort({_id:-1}).limit(20) 1.5 ms 2.0 ms
|
||||
find({}, {proj}).limit(1000) 3.8 ms 4.3 ms
|
||||
aggregate $group by k 7.4 ms 15.8 ms
|
||||
updateOne({_id}) ×50 0.18 ms 0.21 ms
|
||||
updateMany({k: 7}, {$inc}) 1.9 ms 5.9 ms
|
||||
deleteOne({_id}) + insertOne 0.67 ms 5.1 ms
|
||||
node client RSS 163 MB 155 MB
|
||||
|
||||
[concurrency]
|
||||
clients 1 8373 218 38.4x
|
||||
clients 4 20880 491 42.5x
|
||||
clients 8 26349 964 27.3x
|
||||
clients 16 29594 1710 17.3x
|
||||
clients 32 32235 3389 9.5x
|
||||
|
||||
[meta]
|
||||
mfdb_rss_mb 1059
|
||||
md_rss_mb 1242
|
||||
mfdb_reopen 0.3s
|
||||
md_reopen 1.3s
|
||||
mfdb_disk_mb 20MB
|
||||
md_disk_mb 87MB
|
||||
36
tests/e2e/results/bench-20260803-225322.txt
Normal file
36
tests/e2e/results/bench-20260803-225322.txt
Normal file
@@ -0,0 +1,36 @@
|
||||
# multiforadb vs MongoDB benchmark
|
||||
# date: 2026-08-03T19:54:54Z git: 4b70ce6+dirty
|
||||
# args: size=1g doc-size=16k wc=j clients=1 4 8 16 32 per-client=2000
|
||||
# reproduce: bash tests/e2e/bench-run.sh 1g 16k "1 4 8 16 32"
|
||||
|
||||
[main]
|
||||
insertOne (sequential) ×200 0.21 ms 4.4 ms
|
||||
bulk insert throughput 550.4 MB/s 721.6 MB/s
|
||||
docs loaded 65,536 65,536
|
||||
createIndex({k: 1}) 28.9 ms 72.5 ms
|
||||
countDocuments({}) 3.6 ms 11.3 ms
|
||||
findOne({_id: <ObjectId>}) 0.80 ms 0.76 ms
|
||||
findOne({k: 500}) (indexed) 0.73 ms 5.4 ms
|
||||
find({p: {$gte,$lt}}).count() (scan) 14.5 ms 15.4 ms
|
||||
find({}).sort({_id:-1}).limit(20) 1.9 ms 2.6 ms
|
||||
find({}, {proj}).limit(1000) 3.8 ms 4.9 ms
|
||||
aggregate $group by k 11.2 ms 15.5 ms
|
||||
updateOne({_id}) ×50 0.17 ms 0.21 ms
|
||||
updateMany({k: 7}, {$inc}) 2.0 ms 6.1 ms
|
||||
deleteOne({_id}) + insertOne 0.69 ms 4.9 ms
|
||||
node client RSS 150 MB 156 MB
|
||||
|
||||
[concurrency]
|
||||
clients 1 7709 229 33.7x
|
||||
clients 4 20122 493 40.8x
|
||||
clients 8 26210 977 26.8x
|
||||
clients 16 27794 1895 14.7x
|
||||
clients 32 32817 3249 10.1x
|
||||
|
||||
[meta]
|
||||
mfdb_rss_mb 1060
|
||||
md_rss_mb 1178
|
||||
mfdb_reopen 0.3s
|
||||
md_reopen 1.3s
|
||||
mfdb_disk_mb 20MB
|
||||
md_disk_mb 85MB
|
||||
@@ -1,36 +1,36 @@
|
||||
# mongo-lite vs MongoDB benchmark
|
||||
# date: 2026-08-03T08:53:32Z git: c8d547f+dirty
|
||||
# multiforadb vs MongoDB benchmark
|
||||
# date: 2026-08-03T19:54:54Z git: 4b70ce6+dirty
|
||||
# args: size=1g doc-size=16k wc=j clients=1 4 8 16 32 per-client=2000
|
||||
# reproduce: bash tests/e2e/bench-run.sh 1g 16k "1 4 8 16 32"
|
||||
|
||||
[main]
|
||||
insertOne (sequential) ×200 0.20 ms 14.8 ms
|
||||
bulk insert throughput 732.4 MB/s 546.2 MB/s
|
||||
insertOne (sequential) ×200 0.21 ms 4.4 ms
|
||||
bulk insert throughput 550.4 MB/s 721.6 MB/s
|
||||
docs loaded 65,536 65,536
|
||||
createIndex({k: 1}) 74.4 ms 83.6 ms
|
||||
countDocuments({}) 3.1 ms 12.4 ms
|
||||
findOne({_id: <ObjectId>}) 0.59 ms 0.61 ms
|
||||
findOne({k: 500}) (indexed) 0.56 ms 0.94 ms
|
||||
find({p: {$gte,$lt}}).count() (scan) 13.0 ms 15.0 ms
|
||||
find({}).sort({_id:-1}).limit(20) 2.2 ms 2.2 ms
|
||||
find({}, {proj}).limit(1000) 3.5 ms 4.4 ms
|
||||
aggregate $group by k 8.3 ms 13.2 ms
|
||||
updateOne({_id}) ×50 0.14 ms 0.20 ms
|
||||
updateMany({k: 7}, {$inc}) 1.9 ms 6.4 ms
|
||||
deleteOne({_id}) + insertOne 0.58 ms 4.9 ms
|
||||
node client RSS 158 MB 157 MB
|
||||
createIndex({k: 1}) 28.9 ms 72.5 ms
|
||||
countDocuments({}) 3.6 ms 11.3 ms
|
||||
findOne({_id: <ObjectId>}) 0.80 ms 0.76 ms
|
||||
findOne({k: 500}) (indexed) 0.73 ms 5.4 ms
|
||||
find({p: {$gte,$lt}}).count() (scan) 14.5 ms 15.4 ms
|
||||
find({}).sort({_id:-1}).limit(20) 1.9 ms 2.6 ms
|
||||
find({}, {proj}).limit(1000) 3.8 ms 4.9 ms
|
||||
aggregate $group by k 11.2 ms 15.5 ms
|
||||
updateOne({_id}) ×50 0.17 ms 0.21 ms
|
||||
updateMany({k: 7}, {$inc}) 2.0 ms 6.1 ms
|
||||
deleteOne({_id}) + insertOne 0.69 ms 4.9 ms
|
||||
node client RSS 150 MB 156 MB
|
||||
|
||||
[concurrency]
|
||||
clients 1 8435 232 36.4x
|
||||
clients 4 22480 491 45.8x
|
||||
clients 8 25920 966 26.8x
|
||||
clients 16 31079 1842 16.9x
|
||||
clients 32 34041 3632 9.4x
|
||||
clients 1 7709 229 33.7x
|
||||
clients 4 20122 493 40.8x
|
||||
clients 8 26210 977 26.8x
|
||||
clients 16 27794 1895 14.7x
|
||||
clients 32 32817 3249 10.1x
|
||||
|
||||
[meta]
|
||||
ml_rss_mb 553
|
||||
md_rss_mb 1486
|
||||
ml_reopen 0.8s
|
||||
mfdb_rss_mb 1060
|
||||
md_rss_mb 1178
|
||||
mfdb_reopen 0.3s
|
||||
md_reopen 1.3s
|
||||
ml_disk_mb 97MB
|
||||
md_disk_mb 106MB
|
||||
mfdb_disk_mb 20MB
|
||||
md_disk_mb 85MB
|
||||
|
||||
156
tests/e2e/results/m0-gates.txt
Normal file
156
tests/e2e/results/m0-gates.txt
Normal file
@@ -0,0 +1,156 @@
|
||||
# M0 gate results — mmap + WAL storage foundation (PLAN D7)
|
||||
#
|
||||
# Machine: Apple Silicon, macOS 25.5.0, 16 KiB system pages, APFS.
|
||||
# Server: MultiforaDB at the commit named below, ReleaseFast.
|
||||
# Driver: mongodb@7.5.0 (pinned in tests/e2e/package-lock.json).
|
||||
# mongod: 8.3.7, for the parity rows.
|
||||
#
|
||||
# Read this alongside PLAN.md §5. Every number here is reproducible with the
|
||||
# command printed under it; where a gate was not met, the number is recorded as
|
||||
# measured and the reason is stated rather than the workload being tuned until
|
||||
# it passed.
|
||||
|
||||
[D7.1] unit tests, ReleaseFast and ReleaseSafe
|
||||
zig build test -Doptimize=ReleaseFast 122/122 pass
|
||||
zig build test -Doptimize=ReleaseSafe 122/122 pass
|
||||
zig build fuzz fuzz_split, spill, spill2, stress clean
|
||||
Both modes matter: `protect_stable` (the mprotect belt over the published
|
||||
image) is comptime-off in ReleaseFast, and `std.posix.mprotect` not existing
|
||||
in Zig 0.16 was a ReleaseSafe-only compile error.
|
||||
|
||||
[D7.2] end-to-end suites — all green
|
||||
reproduce: zig-out/bin/multiforadb --port 27020 --db /tmp/mfdb-e2e.log \
|
||||
--ttl-sweep-secs 1 & then node tests/e2e/<suite>
|
||||
e2e.js CRUD, operators, aggregate, errors 35/35
|
||||
e2e3.js secondary indexes: unique/sparse/compound 16/16
|
||||
e2e4.js TTL indexes: expiry and rejected specs 17/17
|
||||
e2e5.js miscellaneous command surface 3/3
|
||||
e2e2.js concurrent, 4 writers + 4 readers 2/2
|
||||
e2e2.js crash-a / crash-b (kill -9, then verify) 1/1 then 3/3
|
||||
e2e6.js compaction and log growth (own server) 72/72
|
||||
|
||||
[D7.3] large-collection smoke — big.js
|
||||
reproduce: node tests/e2e/big.js --size 20g --doc-size 16k
|
||||
20g run 4g run
|
||||
documents 1,310,720 262,144 (x 16 KiB)
|
||||
live data 21.47 GB 4.29 GB
|
||||
data file 21.75 GB 4.35 GB (+1.3%, +1.4%)
|
||||
log file, min -> final 16 B -> 2.5 MB 16 B -> 7.7 MB
|
||||
bulk insert 411.0 MB/s 560.3 MB/s
|
||||
peak server RSS during load 18,492 MB 3,318 MB
|
||||
kill -9 then reopen 0.5 s 0.5 s
|
||||
RSS after reopen 237 MB (1.1%) 130 MB (3.0%)
|
||||
count after restart 1,310,720 OK 262,144 OK
|
||||
last doc intact after restart payload 16254B OK
|
||||
kill -9 after 200 acked writes 200/200 200/200
|
||||
This is the milestone's central claim, measured: an open costs the working
|
||||
set, not the size of the database, and it no longer replays the whole log.
|
||||
Reopen is flat at 0.5 s from 4 GB to 21.5 GB, and RSS after reopen falls as a
|
||||
*share* of the data as the database grows — which is what "RSS = working set"
|
||||
has to mean to be worth anything.
|
||||
Before M0 the same shape of run reported 523 MB resident after reopen for a
|
||||
512 MB database, because recovering each document's `_id` meant reading every
|
||||
document at open. That scan is what the mmap foundation deleted.
|
||||
Two other things the 20g run shows, honestly. Peak RSS during the *load* is
|
||||
18.5 GB: writing 21 GB touches 21 GB of pages and the kernel keeps them until
|
||||
it needs the memory, so a bulk load is not where the mmap win shows up. And a
|
||||
cold full scan of 21 GB costs ~55 s — countDocuments, the range scan and the
|
||||
unindexed updateMany all land there, ~390 MB/s off the disk — which is correct
|
||||
and is the reason M1's cursors and index-only paths matter.
|
||||
|
||||
[D7.4] churn gate — doc/slab garbage (PLAN amendment A2 retargets D6.2 here)
|
||||
40,000 x 16 KiB documents (655 MB live), one secondary index, steady state
|
||||
measured as data-file size / live data after the ratio stopped moving.
|
||||
before the fixes after
|
||||
delete half and refill, 6 rounds 4.10x, climbing 1.65x, flat
|
||||
random $set over 5x the collection 3.58x, climbing 2.47x, flat
|
||||
"Climbing" is the whole finding: nothing was being reclaimed at all. Three
|
||||
bugs, all fixed in `db/pager: reclaim what churn abandons` — a compaction
|
||||
trigger that had read `log.data_bytes` since before the log was truncated at
|
||||
every checkpoint, a numeric stable mark that made recycled pages look
|
||||
published, and a first-fit free list that let one-page requests dismantle
|
||||
the extents. A fourth fix (a rebuild publishes twice, so the space it frees
|
||||
is reusable immediately) took the interleaved case from 3.58x to 2.47x.
|
||||
The gate hoped for ~1.3x and this is above it, structurally: a rebuild needs
|
||||
a whole second copy of the live data before the first can be freed, and
|
||||
two-generation retention holds the old copy through two more publishes. The
|
||||
gate's purpose was to decide whether doc-level free lists are needed after
|
||||
M0. They are; that is an M1 item. What M0 owes is a bound, and there is one.
|
||||
|
||||
[D7.5] benchmark parity vs the pre-mmap engine
|
||||
reproduce: bash tests/e2e/bench-run.sh 1g 16k "1 4 8 16 32"
|
||||
baseline: tests/e2e/results/phase8.txt (c8d547f, all-in-RAM engine)
|
||||
M0: three runs, tests/e2e/results/bench-2026080{3-223947,3-225128,3-225322}.txt
|
||||
Read the range, not a single delta. Two consecutive runs of the *same* binary
|
||||
moved the sub-10 ms rows by 27-51% on this machine, so any one comparison
|
||||
reads whatever the noise did that minute. Best-of-three against the baseline:
|
||||
row pre-mmap M0 min..max best
|
||||
insertOne (sequential) x200 0.20 ms 0.20..0.22 ms +0%
|
||||
bulk insert throughput 732.4 MB/s 463.4..555.5 MB/s -24%
|
||||
createIndex({k: 1}) 74.4 ms 27.9..28.9 ms -62%
|
||||
countDocuments({}) 3.1 ms 2.6..3.6 ms -16%
|
||||
findOne({_id: <ObjectId>}) 0.59 ms 0.60..0.80 ms +2%
|
||||
findOne({k: 500}) (indexed) 0.56 ms 0.54..0.73 ms -4%
|
||||
find({p: {$gte,$lt}}).count() (scan) 13.0 ms 11.2..14.5 ms -14%
|
||||
find({}).sort({_id:-1}).limit(20) 2.2 ms 1.5..1.9 ms -32%
|
||||
find({}, {proj}).limit(1000) 3.5 ms 3.6..3.8 ms +3%
|
||||
aggregate $group by k 8.3 ms 7.4..11.2 ms -11%
|
||||
updateOne({_id}) x50 0.14 ms 0.15..0.18 ms +7%
|
||||
updateMany({k: 7}, {$inc}) 1.9 ms 1.9..2.0 ms +0%
|
||||
deleteOne({_id}) + insertOne 0.58 ms 0.63..0.69 ms +9%
|
||||
node client RSS 158 MB 150..163 MB -5%
|
||||
concurrent durable insertOne, docs/s:
|
||||
clients 1 8,435 7,709..8,373 -1%
|
||||
clients 4 22,480 20,122..20,880 -7%
|
||||
clients 8 25,920 26,210..26,349 +2%
|
||||
clients 16 31,079 27,794..29,594 -5%
|
||||
clients 32 34,041 32,235..32,817 -4%
|
||||
One reproducible regression: bulk insert, -24%, steady across all three runs.
|
||||
This is PLAN risk 1 exactly as written -- document bytes now reach the disk
|
||||
uncompressed in the data file on top of the LZ4 log, and that writeback
|
||||
bandwidth is new. Every other row is inside the run-to-run spread. The gate
|
||||
as stated was "no phase8 row regressed"; it is met for the read and latency
|
||||
rows and not for bulk load, which is the trade the milestone makes and which
|
||||
risk 1 anticipated. Untried mitigations, in the order worth trying:
|
||||
MADV_HUGEPAGE / a larger growth chunk, and not writing doc bytes twice
|
||||
(log and slab) for a bulk path that could log an extent reference instead.
|
||||
createIndex at -62% is the other reproducible number, and it comes from the
|
||||
same change: a bulk build now packs pages in the mapping instead of growing
|
||||
an ArrayList.
|
||||
|
||||
[D7.6] MongoDB spec-test scorecard
|
||||
reproduce: bash tests/spec/fetch.sh && node tests/spec/run.js --scorecard
|
||||
total 131 pass 161 fail 195 skip 175 files 0 errored
|
||||
Byte-identical to the scorecard recorded before the storage rewrite. That is
|
||||
the intended result and it is worth stating plainly: M0 replaced the document
|
||||
store, the node arena and the overflow slab, added copy-on-write, a watermark
|
||||
and a checkpoint, and moved every index leaf's payload -- and changed no
|
||||
observable CRUD or aggregate semantics.
|
||||
|
||||
# WHAT THE GATES FOUND
|
||||
# Five bugs, none of which any unit test or e2e suite had reached:
|
||||
# 1. The compaction trigger had been dead since commit 14 (log truncation
|
||||
# zeroes the counter it gated on). Nothing reclaimed doc-slab garbage.
|
||||
# 2. `page_mut_cow` asked `p >= stable_pages`, which is false for a recycled
|
||||
# page, so every write to one copied and freed it again.
|
||||
# 3. First fit let copy-on-write's one-page requests shave the extents the
|
||||
# doc slab needs, so the free list drained every generation and the file
|
||||
# grew anyway.
|
||||
# 4. A rebuild's freed space took two unrelated checkpoints to become
|
||||
# reusable, so the next rebuild grew the file instead of reusing it.
|
||||
# 5. `reserve_pages`' promise was a single counter on the pager. Two upserts
|
||||
# on different collections release it independently, so the first to
|
||||
# finish revoked the second's promise mid-write -- aborting the server at
|
||||
# four concurrent clients, on the first concurrent benchmark since the
|
||||
# data file landed. PLAN risk 3, whose mitigation was never built.
|
||||
# Four of the five are invisible without a workload that runs long enough to
|
||||
# reach a steady state, or wide enough to have two writers. That is the
|
||||
# argument for keeping both the churn gate and the concurrent benchmark in the
|
||||
# gate list rather than treating them as optional.
|
||||
#
|
||||
# ONE COMPATIBILITY GAP FOUND, NOT FIXED (out of M0's scope, storage-only)
|
||||
# `update.apply` (src/update.zig:18) rejects any update document whose first
|
||||
# key is not `$`, so replacement-style writes -- replaceOne, findOneAndReplace,
|
||||
# bulkWrite's replaceOne -- fail with "bad update". Found while building the
|
||||
# churn workload. It is a CRUD feature, not storage, and it is already part of
|
||||
# what the 161 spec failures cover.
|
||||
Reference in New Issue
Block a user