Aleksey Shakhmatov
61fe952125
Replace Index.entries (one sorted array) with a B+tree so writes into an already-built index stop being quadratic. Nodes are fixed 4 KiB slotted pages in a flat u32-addressed ArrayListUnmanaged(Node); records longer than a quarter page spill to an append-only overflow slab (BSON strings reach 16 MB). Leaves are doubly linked for ordered iteration; the flat node array stays one contiguous byte range for a later checkpoint. Insertion descends by separator and splits leaves/internals upward, promoting keys via a stable copy (a nested split can otherwise clobber the promoted-key scratch). Deletion does not rebalance: emptied leaves are unlinked and dropped from their parent, internal nodes may carry one child, and dead pages are abandoned in place (node memory peaks at the tree's peak size, exactly what the old array's capacity did). Lookups are lower-bound seeks plus leaf-chain band scans, so equal keys may span leaves freely. Bulk build (append_doc_entries + finish_bulk) sorts a staging array and packs leaves bottom-up. reserve_for now takes the built entries and reserves exact overflow bytes plus a worst-case node count, keeping insert_entries infallible after the log append. db.zig: TTL sweep now seeks the minimum-datetime encoded key and walks the contiguous datetime band, stopping at the cutoff or type change. Measured (tests/e2e/results/phase2.txt): updateMany 17.3 -> 1.8 ms (2.8x slower than MongoDB -> 3.7x faster), createIndex 62 -> 51 ms. Verified: unit suite ReleaseFast/ReleaseSafe/Debug (incl. the existing lookup_range and remove_doc differentials, plus a new incremental insert/remove differential against a brute-force model), the crash pair, e2e3/e2e4/e2e6, and dev stress tests for depth-2 splits, full drains, and spilled records through internal levels.
End-to-end tests with the official MongoDB Node.js driver
These exercise mongo-lite from a real driver over TCP: full CRUD, query operators, aggregation, error codes, concurrent clients, crash recovery, and the whole lifecycle including server restarts.
Setup
cd tests/e2e
npm init -y >/dev/null
npm install mongodb
Run
Most suites expect a server running on port 27020:
zig build
zig-out/bin/mongo-lite --port 27020 --db /tmp/ml-e2e.log --ttl-sweep-secs 1 &
node tests/e2e/e2e.js # CRUD + operators + aggregate + errors (29 checks)
node tests/e2e/e2e2.js concurrent # 8 clients: 4 writers + 4 readers (2 checks)
node tests/e2e/e2e2.js crash-a # write 50 docs, then kill -9 the server
node tests/e2e/e2e2.js crash-b # restart and verify all 50 survived
node tests/e2e/e2e3.js # secondary indexes: unique/sparse/compound (16 checks)
node tests/e2e/e2e4.js # TTL indexes: expiry + rejected specs (15 checks)
e2e4.js needs the server started with --ttl-sweep-secs 1 (the default is
60 seconds); the other suites do not care about the flag.
e2e6.js is the full-lifecycle suite and is self-contained: it spawns its
own server on port 27220 with a fresh log, runs the whole feature surface,
restarts the server twice (graceful SIGTERM, then kill -9 mid-write) and
verifies everything survived:
node tests/e2e/e2e6.js # 73 checks, ~15 s, needs no running server
E2E6_PORT=27300 node tests/e2e/e2e6.js # different port if 27220 is taken
Rebuild with zig build after any change under src/ before restarting the
server: zig build test compiles the test binary only and leaves
zig-out/bin/mongo-lite stale, so the suites keep running against the old
rules and report failures that the source no longer explains.
e2e2.js concurrent is safe to repeat against a running server (it drops its
collection first); crash-a/crash-b are two halves of one scenario.
Multi-GB collections: big.js
big.js is a load harness, not a pass/fail suite: it spawns a server, bulk
loads up to ~5 GB, and reports insert throughput, the compaction behavior,
server RSS, per-operation latencies, reopen (replay) time, and kill -9
durability.
node tests/e2e/big.js --quick # 268 MB smoke run
node tests/e2e/big.js --size 5g --doc-size 128k --oid --batch 200 \
--compact-threshold 2g # ~5 GB, 40k docs
Options: --size/--doc-size/--batch (k/m/g suffixes), --oid
(ObjectId _ids — see below), --index <field> (secondary index before
loading), --compact-threshold <bytes> (passed to the server),
--port, --keep (keep the db file).
Measured behavior (all documented in the top-level README):
- Build in ReleaseFast —
zig builddefaults to it; a Debug server is 10-200x slower on every path. - Insert throughput collapses under the default 16 MiB compaction
threshold: every ~16 MB of writes rewrites the whole log with one fsync
per record (O(n²) total). With
--compact-threshold 2gthe rate stays flat (hundreds of MB/s at 128 KB docs in ReleaseFast). Raise the threshold for bulk loads. findOne({_id})is O(1) only for ObjectId_ids. Integer_ids are serialization-ambiguous (int32/int64/double compare equal but hash differently), so the docs-map fast path is skipped and every lookup is a full scan. Use the driver's default ObjectId ids on big collections.- The engine holds everything in RAM: ~1-1.2x the data size at 128 KB docs (more at 16 KB docs, where per-document arena overhead dominates). A 5 GB collection needs roughly 6-7 GB of RAM.
- Reopen of a 5 GB log replays in ~10 s (ReleaseFast); every committed write survives kill -9.
Comparing against real MongoDB: compare.js + compare-run.sh
bash tests/e2e/compare-run.sh [size] [doc-size] # e.g. 1g 16k
Starts mongod (brew install mongodb-community) on :27018 and mongo-lite
on :27019, runs the same driver workload against each (durable writes:
mongo-lite fsyncs per command, mongod runs with j: true), measures kill -9
reopen for both, and prints a side-by-side table. compare.js alone runs
one side (see its --help-style header comment).