Aleksey Shakhmatov
504179acd1
PLAN D7's six items, with the numbers and the command that reproduces each in
tests/e2e/results/m0-gates.txt. Unit tests green in both optimize modes, the
whole e2e matrix green, the spec scorecard byte-identical at 131/161/195, and
the large smoke run at the scale D7.3 asked for:
21.47 GB collection (1,310,720 x 16 KiB)
data file 21.75 GB (+1.3% over the documents)
log after the load 2.5 MB (checkpoints reclaim it)
kill -9 then reopen 0.5 s (0.5 s at 4 GB too -- flat)
RSS after reopen 237 MB (1.1% of the data)
count after restart 1,310,720 last document byte-intact
acked writes after kill 200/200
That is the milestone's claim, measured: an open costs the working set rather
than the size of the database. Before M0 the same measurement was 523 MB
resident for a 512 MB database, because recovering each document's `_id` meant
reading every document at open.
Two gates need reading rather than a tick, and m0-gates.txt says so where a
reader would otherwise take a tick for granted.
The churn gate settles at 1.65x live data (delete-heavy) to 2.47x
(update-heavy), flat, above the ~1.3x amendment A2 hoped for. Rebuild-only
reclamation cannot reach that: it needs a whole second copy of the live data
before the first can be freed. The gate existed to decide whether doc-level
free lists are needed after M0, and that is the answer.
Benchmark parity holds for every read and latency row inside the run-to-run
spread, and bulk insert regresses 24% (732 -> 555 MB/s), reproducibly across
three runs. Risk 1 as written: document bytes now reach the disk uncompressed
on top of the LZ4 log. createIndex improves 62% from the same change.
Three measurement bugs fixed while running the gates, because each would have
put a false number in the README:
- `compare-run.sh` measured "db on disk" as `du` of the log alone against
`du` of mongod's whole dbpath. It reported 20 MB for a 1 GB collection --
the documents had moved to <db>.data. Honest figure, measured: 914 MB of
allocated blocks against mongod's compressed 85 MB.
- `big.js` counted "compaction events" as "the log shrank", which is a
*checkpoint* now. It claimed 12 compaction rewrites during a pure insert
load, which has no garbage to compact.
- `big.js` labelled peak RSS "in-memory engine: docs live in RAM" and its
summary said the collection was held "fully in RAM". Both were true of the
engine this milestone replaced.
README: the storage section described an all-in-RAM engine; the comparison
table mixed one old run's body with three new rows; and `findOne({_id})` was
documented as a full scan for integer ids, which the ordered `_id_` index made
false (2 ms against 55 s for a scan of the same 21.5 GB collection). The table
is now best-of-three for both servers, with the measured variance stated, since
two runs of the same binary moved the sub-10 ms rows by 27-51%.
End-to-end tests with the official MongoDB Node.js driver
These exercise MultiforaDB from a real driver over TCP: full CRUD, query operators, aggregation, error codes, concurrent clients, crash recovery, and the whole lifecycle including server restarts.
Setup
cd tests/e2e
npm init -y >/dev/null
npm install mongodb
Run
Most suites expect a server running on port 27020:
zig build
zig-out/bin/multiforadb --port 27020 --db /tmp/mfdb-e2e.log --ttl-sweep-secs 1 &
node tests/e2e/e2e.js # CRUD + operators + aggregate + errors (29 checks)
node tests/e2e/e2e2.js concurrent # 8 clients: 4 writers + 4 readers (2 checks)
node tests/e2e/e2e2.js crash-a # write 50 docs, then kill -9 the server
node tests/e2e/e2e2.js crash-b # restart and verify all 50 survived
node tests/e2e/e2e3.js # secondary indexes: unique/sparse/compound (16 checks)
node tests/e2e/e2e4.js # TTL indexes: expiry + rejected specs (15 checks)
e2e4.js needs the server started with --ttl-sweep-secs 1 (the default is
60 seconds); the other suites do not care about the flag.
e2e6.js is the full-lifecycle suite and is self-contained: it spawns its
own server on port 27220 with a fresh log, runs the whole feature surface,
restarts the server twice (graceful SIGTERM, then kill -9 mid-write) and
verifies everything survived:
node tests/e2e/e2e6.js # 73 checks, ~15 s, needs no running server
E2E6_PORT=27300 node tests/e2e/e2e6.js # different port if 27220 is taken
Rebuild with zig build after any change under src/ before restarting the
server: zig build test compiles the test binary only and leaves
zig-out/bin/multiforadb stale, so the suites keep running against the old
rules and report failures that the source no longer explains.
e2e2.js concurrent is safe to repeat against a running server (it drops its
collection first); crash-a/crash-b are two halves of one scenario.
Multi-GB collections: big.js
big.js is a load harness, not a pass/fail suite: it spawns a server, bulk
loads up to ~5 GB, and reports insert throughput, the compaction behavior,
server RSS, per-operation latencies, reopen (replay) time, and kill -9
durability.
node tests/e2e/big.js --quick # 268 MB smoke run
node tests/e2e/big.js --size 5g --doc-size 128k --oid --batch 200 \
--compact-threshold 2g # ~5 GB, 40k docs
Options: --size/--doc-size/--batch (k/m/g suffixes), --oid
(ObjectId _ids — see below), --index <field> (secondary index before
loading), --compact-threshold <bytes> (passed to the server),
--port, --keep (keep the db file).
Measured behavior (all documented in the top-level README):
- Build in ReleaseFast —
zig builddefaults to it; a Debug server is 10-200x slower on every path. - Insert throughput collapses under the default 16 MiB compaction
threshold: every ~16 MB of writes rewrites the whole log with one fsync
per record (O(n²) total). With
--compact-threshold 2gthe rate stays flat (hundreds of MB/s at 128 KB docs in ReleaseFast). Raise the threshold for bulk loads. findOne({_id})is O(1) only for ObjectId_ids. Integer_ids are serialization-ambiguous (int32/int64/double compare equal but hash differently), so the docs-map fast path is skipped and every lookup is a full scan. Use the driver's default ObjectId ids on big collections.- The engine holds everything in RAM: ~1-1.2x the data size at 128 KB docs (more at 16 KB docs, where per-document arena overhead dominates). A 5 GB collection needs roughly 6-7 GB of RAM.
- Reopen of a 5 GB log replays in ~10 s (ReleaseFast); every committed write survives kill -9.
Comparing against real MongoDB: compare.js + compare-run.sh
bash tests/e2e/compare-run.sh [size] [doc-size] # e.g. 1g 16k
Starts mongod (brew install mongodb-community) on :27018 and MultiforaDB
on :27019, runs the same driver workload against each (durable writes:
MultiforaDB fsyncs per command, mongod runs with j: true), measures kill -9
reopen for both, and prints a side-by-side table. compare.js alone runs
one side (see its --help-style header comment).
Iteration-to-iteration comparison: bench-run.sh + concurrent.js
bash tests/e2e/bench-run.sh [size] [doc-size] ["clients..."] # e.g. 1g 16k "1 4 8 16 32"
Runs the main suite (compare-run.sh) plus a concurrent durable-write
comparison (concurrent.js, N clients each doing sequential insertOne
with {w:1, j:true} — the group-commit path under real contention), then
writes a machine-readable, versioned report to
tests/e2e/results/bench-<timestamp>.txt and prints a diff of the
MultiforaDB numbers against the previous run (results/bench-latest.txt).
The report has [main] / [concurrency] / [meta] sections with
name<TAB>value rows; bench-run.sh 1g 16k reproduces the phase8 gate
(see results/phase8.txt).