dev/MultiforaDB
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

tests/e2e: iteration-to-iteration benchmark harness

Browse Source 
compare-run.sh answers "how do we compare to MongoDB"; it says nothing
about whether a change made things better or worse than last week. Add a
harness that records each run and diffs it against the previous one.

bench-run.sh wraps compare-run.sh, adds a concurrent durable-write
comparison (concurrent.js: N clients each doing sequential insertOne with
{w:1, j:true}, exercising the group-commit path under real contention),
writes a versioned name<TAB>value report to results/bench-<timestamp>.txt,
and prints a diff of our numbers against results/bench-latest.txt.

Also:
- compare-run.sh polled with fixed sleeps, which are flaky once earlier
  benchmark phases have warmed the machine; both servers now wait on a
  real driver connection instead.
- a dispatch error only reached the client as a generic InternalError,
  with nothing on the server side naming the failing command; log the
  connection, command and error name before replacing the reply.
This commit is contained in:
Alexey Shakhmatov
2026-08-03 12:33:28 +03:00
parent 720540860a
commit ac464f2b92
10 changed files with 441 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

34
tests/e2e/results/bench-20260803-112548.txt Normal file
View File

@@ -0,0 +1,34 @@
# mongo-lite vs MongoDB benchmark
# date: 2026-08-03T08:26:46Z git: c8d547f+dirty
# args: size=128m doc-size=8k wc=j clients=1 2 4 per-client=2000
# reproduce: bash tests/e2e/bench-run.sh 128m 8k "1 2 4"
[main]
insertOne (sequential) ×200 0.20 ms 5.9 ms
bulk insert throughput 664.4 MB/s 357.9 MB/s
docs loaded 16,384 16,384
createIndex({k: 1}) 10.4 ms 45.2 ms
countDocuments({}) 0.73 ms 8.6 ms
findOne({_id: <ObjectId>}) 0.35 ms 0.62 ms
findOne({k: 500}) (indexed) 0.36 ms 1.2 ms
find({p: {$gte,$lt}}).count() (scan) 2.4 ms 3.2 ms
find({}).sort({_id:-1}).limit(20) 2.0 ms 2.2 ms
find({}, {proj}).limit(1000) — 4.6 ms
aggregate $group by k — 4.6 ms
updateOne({_id}) ×50 — 0.26 ms
updateMany({k: 7}, {$inc}) — 5.7 ms
deleteOne({_id}) + insertOne — 4.8 ms
node client RSS — 136 MB
[concurrency]
clients 1 8117 175 46.4x
clients 2 17102 230 74.4x
clients 4 21967 352 62.4x
[meta]
ml_rss_mb 158
md_rss_mb 251
ml_reopen 0.3s
md_reopen 0.4s
ml_disk_mb 7MB
md_disk_mb 7MB

36
tests/e2e/results/bench-20260803-113439.txt Normal file
View File

@@ -0,0 +1,36 @@
# mongo-lite vs MongoDB benchmark
# date: 2026-08-03T08:36:09Z git: c8d547f+dirty
# args: size=1g doc-size=16k wc=j clients=1 4 8 16 32 per-client=2000
# reproduce: bash tests/e2e/bench-run.sh 1g 16k "1 4 8 16 32"
[main]
insertOne (sequential) ×200 0.20 ms 4.9 ms
bulk insert throughput 748.8 MB/s 685.2 MB/s
docs loaded 65,536 65,536
createIndex({k: 1}) 79.8 ms 86.6 ms
countDocuments({}) 3.4 ms 13.6 ms
findOne({_id: <ObjectId>}) 0.71 ms 0.61 ms
findOne({k: 500}) (indexed) 0.66 ms 2.0 ms
find({p: {$gte,$lt}}).count() (scan) 13.5 ms 12.4 ms
find({}).sort({_id:-1}).limit(20) 2.3 ms 2.6 ms
find({}, {proj}).limit(1000) 3.6 ms 4.4 ms
aggregate $group by k 9.5 ms 13.7 ms
updateOne({_id}) ×50 0.14 ms 0.18 ms
updateMany({k: 7}, {$inc}) 2.1 ms 6.1 ms
deleteOne({_id}) + insertOne 0.73 ms 5.0 ms
node client RSS 160 MB 153 MB
[concurrency]
clients 1 8816 203 43.4x
clients 4 22142 472 46.9x
clients 8 27716 967 28.7x
clients 16 32031 1858 17.2x
clients 32 3759 0.0x
[meta]
ml_rss_mb 547
md_rss_mb 1317
ml_reopen 0.8s
md_reopen 1.3s
ml_disk_mb 97MB
md_disk_mb 89MB

36
tests/e2e/results/bench-20260803-115200.txt Normal file
View File

@@ -0,0 +1,36 @@
# mongo-lite vs MongoDB benchmark
# date: 2026-08-03T08:53:32Z git: c8d547f+dirty
# args: size=1g doc-size=16k wc=j clients=1 4 8 16 32 per-client=2000
# reproduce: bash tests/e2e/bench-run.sh 1g 16k "1 4 8 16 32"
[main]
insertOne (sequential) ×200 0.20 ms 14.8 ms
bulk insert throughput 732.4 MB/s 546.2 MB/s
docs loaded 65,536 65,536
createIndex({k: 1}) 74.4 ms 83.6 ms
countDocuments({}) 3.1 ms 12.4 ms
findOne({_id: <ObjectId>}) 0.59 ms 0.61 ms
findOne({k: 500}) (indexed) 0.56 ms 0.94 ms
find({p: {$gte,$lt}}).count() (scan) 13.0 ms 15.0 ms
find({}).sort({_id:-1}).limit(20) 2.2 ms 2.2 ms
find({}, {proj}).limit(1000) 3.5 ms 4.4 ms
aggregate $group by k 8.3 ms 13.2 ms
updateOne({_id}) ×50 0.14 ms 0.20 ms
updateMany({k: 7}, {$inc}) 1.9 ms 6.4 ms
deleteOne({_id}) + insertOne 0.58 ms 4.9 ms
node client RSS 158 MB 157 MB
[concurrency]
clients 1 8435 232 36.4x
clients 4 22480 491 45.8x
clients 8 25920 966 26.8x
clients 16 31079 1842 16.9x
clients 32 34041 3632 9.4x
[meta]
ml_rss_mb 553
md_rss_mb 1486
ml_reopen 0.8s
md_reopen 1.3s
ml_disk_mb 97MB
md_disk_mb 106MB

36
tests/e2e/results/bench-latest.txt Normal file
View File

@@ -0,0 +1,36 @@
# mongo-lite vs MongoDB benchmark
# date: 2026-08-03T08:53:32Z git: c8d547f+dirty
# args: size=1g doc-size=16k wc=j clients=1 4 8 16 32 per-client=2000
# reproduce: bash tests/e2e/bench-run.sh 1g 16k "1 4 8 16 32"
[main]
insertOne (sequential) ×200 0.20 ms 14.8 ms
bulk insert throughput 732.4 MB/s 546.2 MB/s
docs loaded 65,536 65,536
createIndex({k: 1}) 74.4 ms 83.6 ms
countDocuments({}) 3.1 ms 12.4 ms
findOne({_id: <ObjectId>}) 0.59 ms 0.61 ms
findOne({k: 500}) (indexed) 0.56 ms 0.94 ms
find({p: {$gte,$lt}}).count() (scan) 13.0 ms 15.0 ms
find({}).sort({_id:-1}).limit(20) 2.2 ms 2.2 ms
find({}, {proj}).limit(1000) 3.5 ms 4.4 ms
aggregate $group by k 8.3 ms 13.2 ms
updateOne({_id}) ×50 0.14 ms 0.20 ms
updateMany({k: 7}, {$inc}) 1.9 ms 6.4 ms
deleteOne({_id}) + insertOne 0.58 ms 4.9 ms
node client RSS 158 MB 157 MB
[concurrency]
clients 1 8435 232 36.4x
clients 4 22480 491 45.8x
clients 8 25920 966 26.8x
clients 16 31079 1842 16.9x
clients 32 34041 3632 9.4x
[meta]
ml_rss_mb 553
md_rss_mb 1486
ml_reopen 0.8s
md_reopen 1.3s
ml_disk_mb 97MB
md_disk_mb 106MB

72
tests/e2e/results/phase8.txt Normal file
View File

@@ -0,0 +1,72 @@
# Phase 8 gate — mongo-lite vs MongoDB 8.3.7, 1g dataset / ~16k docs
# Ratio < 1.0 = mongo-lite faster. Reproduce: bash tests/e2e/bench-run.sh 1g 16k
# Working tree: c8d547f + uncommitted diff (epilogue-commit/group-commit work) +
# three bugs found and fixed while benchmarking (details at the bottom).
# Machine-readable copy: tests/e2e/results/bench-latest.txt (auto-diffed on
# every bench-run.sh invocation).
benchmark mongo-lite mongodb ratio
insertOne (sequential) ×200 0.20 ms 14.8 ms 0.0x
bulk insert throughput 732.4 MB/s 546.2 MB/s 1.3x
docs loaded 65,536 65,536 1.0x
createIndex({k: 1}) 74.4 ms 83.6 ms 0.9x
countDocuments({}) 3.1 ms 12.4 ms 0.3x
findOne({_id: <ObjectId>}) 0.59 ms 0.61 ms 1.0x
findOne({k: 500}) (indexed) 0.56 ms 0.94 ms 0.6x
find({p: {$gte,$lt}}).count() (scan) 13.0 ms 15.0 ms 0.9x
find({}).sort({_id:-1}).limit(20) 2.2 ms 2.2 ms 1.0x
find({}, {proj}).limit(1000) 3.5 ms 4.4 ms 0.8x
aggregate $group by k 8.3 ms 13.2 ms 0.6x
updateOne({_id}) ×50 0.14 ms 0.20 ms 0.7x
updateMany({k: 7}, {$inc}) 1.9 ms 6.4 ms 0.3x
deleteOne({_id}) + insertOne 0.58 ms 4.9 ms 0.1x
node client RSS 158 MB 157 MB 1.0x
server RSS 553 MB 1486 MB
kill -9 reopen 0.8s 1.3s
db on disk 97MB 106MB
# NEW: concurrent durable writes (sequential insertOne per client, {w:1, j:true})
# through the official driver, both servers, same workload:
# clients | mongodb | mongo-lite | ratio
# 1 | 232 | 8,435 | 36x
# 4 | 491 | 22,480 | 46x
# 8 | 966 | 25,920 | 27x
# 16 | 1,842 | 31,079 | 17x
# 32 | 3,632 | 34,041 | 9x
# (mongo-lite scales to ~34k durable docs/s at 32 clients; mongod tops out
# around 3.6k on this machine's fsync-per-write path.)
# THREE BUGS FOUND AND FIXED WHILE BENCHMARKING
# All three were invisible to the single-connection compare.js suite and only
# surfaced under the new concurrent harness (tests/e2e/concurrent.js).
#
# 1. Group-commit deadlock (>=6 concurrent writers froze permanently).
# std.Io.Condition.signal() delivers exactly ONE wakeup. The commit
# leader's completion woke a single follower on `while (committing)` and
# stranded the rest; an append's signal could also be consumed by a
# follower, stranding the leader on the pending_appends drain. Signals
# were also sent without commit_lock, leaving a lost-wakeup window
# between a waiter's epoch snapshot and its futex sleep.
# Fix (src/db.zig): all commit_done signals hold commit_lock and use
# broadcast(). Reproduced on HEAD too (introduced by the group-commit
# work, not the uncommitted diff).
#
# 2. Racy assertion in commit(): assert(pending_appends == 0) after the
# drain loop. A NEW append can start at any moment (the increment takes
# no commit_lock), so the assert fired spuriously under load and crashed
# the server (32 concurrent clients, "commit would seal while appends
# are still in flight"). The seal's real invariant holds via log_lock
# ordering; the instant check was wrong. Fix: removed the assert.
#
# 3. Slab segment-start corruption in Collection.slab_append: a pointer
# `last` into slab.items dangled across the slab.append() that starts the
# next segment (the list can reallocate), so the new segment's start
# offset was computed from freed memory. Docs in the affected segment
# were then located at wrong offsets, surfacing as InvalidBson on reads
# with a batch limit >~50 (or a crash in Debug builds). Position varied
# between runs (heap contents vary). Fix: read the last segment's length
# as a value before the append.
#
# Verification after the fixes: unit suite green, e2e6 72/72 (incl. kill -9
# mid-write recovery), crash pair, concurrent 8-client, indexes, TTL all
# green; concurrent stress 1/4/8/16/32 clients x5 sequential rounds clean.