Aleksey Shakhmatov
75e412a4af
Matching allocated an ArrayList per filter field per candidate document,
on the process-wide allocator, to hold what is almost always a single
value. Candidates now collect into a stack buffer that spills to the heap
only for arrays: measured 15.7 -> 12.0ms on a 65,536-document range scan.
The OOM-propagation test moves with it. Its point is that a failed
collection must surface as an error rather than an empty candidate list,
which would make $ne and $exists:false report a match -- a wrong answer
rather than a failed one. That invariant still holds on the spill path, so
the test now uses an array long enough to reach the allocator, and a new
test pins the flip side: the common single-value match now completes
correctly even when the allocator always fails, because it never calls it.
Query operators were dispatched by a chain of up to fourteen mem.eql per
value per document, with $gt/$gte/$lt/$lte re-comparing the operator name
inside the loop over candidate values. Names resolve to an enum once per
filter field. Command dispatch likewise walked a 30-entry table comparing
strings; it is a comptime StaticStringMap now.
Each request built a fresh reply arena and handed its pages straight back.
One reply per connection, reset between requests, keeps them.
countDocuments() arrives as [{$match: F}?, {$group: {_id: <literal>,
n: {$sum: 1}}}], which the general path answered by materializing every
matching document and discarding them all. It is now recognized and
answered from a counting scan: countDocuments({}) 2.3 -> 1.5ms.
The detector is deliberately conservative -- grouping by "$field", summing
a field, an unmodelled accumulator or any extra stage all fall through to
the general path, since those need the documents themselves. A unit test
pins each accept and reject, and the whole count path was checked against
the general one through the real driver, including the shapes that must
not take it.
The filtered range-scan row does not move: it is bound by walking 65,536
documents that each live in their own arena, not by the matcher. That is
Phase 4 work.
Verified: 77 unit tests under ReleaseFast and ReleaseSafe, e2e
29/16/17/3/2, the crash pair, e2e6 72/72.
End-to-end tests with the official MongoDB Node.js driver
These exercise mongo-lite from a real driver over TCP: full CRUD, query operators, aggregation, error codes, concurrent clients, crash recovery, and the whole lifecycle including server restarts.
Setup
cd tests/e2e
npm init -y >/dev/null
npm install mongodb
Run
Most suites expect a server running on port 27020:
zig build
zig-out/bin/mongo-lite --port 27020 --db /tmp/ml-e2e.log --ttl-sweep-secs 1 &
node tests/e2e/e2e.js # CRUD + operators + aggregate + errors (29 checks)
node tests/e2e/e2e2.js concurrent # 8 clients: 4 writers + 4 readers (2 checks)
node tests/e2e/e2e2.js crash-a # write 50 docs, then kill -9 the server
node tests/e2e/e2e2.js crash-b # restart and verify all 50 survived
node tests/e2e/e2e3.js # secondary indexes: unique/sparse/compound (16 checks)
node tests/e2e/e2e4.js # TTL indexes: expiry + rejected specs (15 checks)
e2e4.js needs the server started with --ttl-sweep-secs 1 (the default is
60 seconds); the other suites do not care about the flag.
e2e6.js is the full-lifecycle suite and is self-contained: it spawns its
own server on port 27220 with a fresh log, runs the whole feature surface,
restarts the server twice (graceful SIGTERM, then kill -9 mid-write) and
verifies everything survived:
node tests/e2e/e2e6.js # 73 checks, ~15 s, needs no running server
E2E6_PORT=27300 node tests/e2e/e2e6.js # different port if 27220 is taken
Rebuild with zig build after any change under src/ before restarting the
server: zig build test compiles the test binary only and leaves
zig-out/bin/mongo-lite stale, so the suites keep running against the old
rules and report failures that the source no longer explains.
e2e2.js concurrent is safe to repeat against a running server (it drops its
collection first); crash-a/crash-b are two halves of one scenario.
Multi-GB collections: big.js
big.js is a load harness, not a pass/fail suite: it spawns a server, bulk
loads up to ~5 GB, and reports insert throughput, the compaction behavior,
server RSS, per-operation latencies, reopen (replay) time, and kill -9
durability.
node tests/e2e/big.js --quick # 268 MB smoke run
node tests/e2e/big.js --size 5g --doc-size 128k --oid --batch 200 \
--compact-threshold 2g # ~5 GB, 40k docs
Options: --size/--doc-size/--batch (k/m/g suffixes), --oid
(ObjectId _ids — see below), --index <field> (secondary index before
loading), --compact-threshold <bytes> (passed to the server),
--port, --keep (keep the db file).
Measured behavior (all documented in the top-level README):
- Build in ReleaseFast —
zig builddefaults to it; a Debug server is 10-200x slower on every path. - Insert throughput collapses under the default 16 MiB compaction
threshold: every ~16 MB of writes rewrites the whole log with one fsync
per record (O(n²) total). With
--compact-threshold 2gthe rate stays flat (hundreds of MB/s at 128 KB docs in ReleaseFast). Raise the threshold for bulk loads. findOne({_id})is O(1) only for ObjectId_ids. Integer_ids are serialization-ambiguous (int32/int64/double compare equal but hash differently), so the docs-map fast path is skipped and every lookup is a full scan. Use the driver's default ObjectId ids on big collections.- The engine holds everything in RAM: ~1-1.2x the data size at 128 KB docs (more at 16 KB docs, where per-document arena overhead dominates). A 5 GB collection needs roughly 6-7 GB of RAM.
- Reopen of a 5 GB log replays in ~10 s (ReleaseFast); every committed write survives kill -9.
Comparing against real MongoDB: compare.js + compare-run.sh
bash tests/e2e/compare-run.sh [size] [doc-size] # e.g. 1g 16k
Starts mongod (brew install mongodb-community) on :27018 and mongo-lite
on :27019, runs the same driver workload against each (durable writes:
mongo-lite fsyncs per command, mongod runs with j: true), measures kill -9
reopen for both, and prints a side-by-side table. compare.js alone runs
one side (see its --help-style header comment).