compare-run.sh answers "how do we compare to MongoDB"; it says nothing
about whether a change made things better or worse than last week. Add a
harness that records each run and diffs it against the previous one.
bench-run.sh wraps compare-run.sh, adds a concurrent durable-write
comparison (concurrent.js: N clients each doing sequential insertOne with
{w:1, j:true}, exercising the group-commit path under real contention),
writes a versioned name<TAB>value report to results/bench-<timestamp>.txt,
and prints a diff of our numbers against results/bench-latest.txt.
Also:
- compare-run.sh polled with fixed sleeps, which are flaky once earlier
benchmark phases have warmed the machine; both servers now wait on a
real driver connection instead.
- a dispatch error only reached the client as a generic InternalError,
with nothing on the server side naming the failing command; log the
connection, command and error name before replacing the reply.
Three defects, each of which made the database lose data that had already
been acknowledged, or answer a client with a malformed reply.
- Engine.commit decided a writer was "already covered" by comparing
log.end_pos with the position of the last completed commit. Under block
framing an append leaves its bytes in the log's open in-memory block and
does not move end_pos -- only sealing does. So once the first commit had
set committed_end = end_pos, every later write command found itself
covered and returned without sealing or syncing anything. A no-op
deleteMany followed by insertMany(50) was acknowledged with the file
still 16 bytes (its header) and lost all 50 documents on kill -9, which
is precisely what e2e2's crash pair does. Coverage is now decided by
sequence number, which counts records rather than bytes on disk.
- Compaction read the new log's end position before syncing it, but the
sync is what seals the open block, and the seal is what moves end_pos
past it. Appends after a compaction therefore started inside the
compacted file's last block and overwrote it, so those documents were
gone at the next replay: e2e6's phase 2 ended with 1000 documents in
memory and 996 after a graceful restart.
- cmd_find returned early on a missing namespace without putting anything
in the reply, so a find on an unknown collection arrived at the driver as
a response with no `ok` field ("MongoServerError: n/a") instead of an
empty cursor. The other commands' missing-namespace paths were fine.
Verified with the unit suite in ReleaseFast/ReleaseSafe/Debug, the split
fuzzer, all six e2e suites (e2e6 back to 72/72) and the kill -9 crash pair
-- none of which passed beforehand -- plus 13 kill -9 runs over 1/2/8
connections with 1200 acknowledged inserts each and nothing lost.
tests/e2e/results/phase7.txt records the benchmark with the fixes in place:
no regression against phase6 (bulk 739 -> 753 MB/s, updateMany 1.9 -> 2.0
ms, RSS 547 -> 546 MB), and concurrent durable writes now measurable at
7.1k/15.0k/21.8k docs/s over 1/8/32 connections.
The single engine-wide reader/writer lock is replaced by a lock hierarchy,
so writes to different collections no longer serialize on one mutex:
- Collections are heap-allocated, so their addresses are stable while a
command holds a collection lock (the maps only store pointers).
- A catalog rwlock guards the database/collection maps: shared for every
command (so a concurrent DDL cannot mutate the maps underneath it),
exclusive for create/drop/dropDatabase. Each collection has its own
rwlock; the ordering is always catalog -> collection -> log lock, never
two collection locks at once (TTL sweep and compaction take collections
one at a time).
- Command dispatch acquires the catalog + target collection locks for the
handler's duration, resolving the collection (creating it for writes)
under the catalog lock; create/drop upgrade to the exclusive catalog lock.
- Appends never fsync. Each write command's epilogue releases the
collection lock, then commits once (seal + fsync) with a leader/follower
group commit: the leader waits for writers mid-append (a pending counter)
so its seal covers them, and followers whose records the seal covered
skip their own fsync. Every acknowledged write is fsynced before its
reply (crash pair verified); an unacknowledged write may vanish and a
reader may observe a write before its fsync — ordinary w:1 j:true
semantics instead of 'the log describes >= memory'.
- Compaction snapshots collections without the log lock (so a concurrent
writer holding one can always finish its append) and retries when a
writer appended mid-snapshot (detected via the record seq), then swaps
under the log lock — no deadlock. The compaction trigger moved to the
command epilogue and the TTL monitor.
- Engine.dup_index moved to the collection (per-command error paths).
Also lands two B-tree edge-case fixes driven by tests that were in flight:
a churned leaf full of dead bytes no longer splits with an empty right
half (the leaf is repacked before splitting, and an emptied node's page is
fully free again), and a slot-count split with all large records on one
side shifts records between the halves until the new record fits. Plus a
randomised fuzz test over key sizes (src/fuzz_split.zig) and the two
regression tests.
Measured (tests/e2e/results/phase6.txt): no regression on the
single-connection benchmark; concurrent durable-insert throughput ~5.1k ->
12.5k docs/s from 1 -> 8 clients, ~14.8k at 32. Verified: unit suite in
all three modes, all e2e suites, the kill -9 crash pair.
Documents live as canonical BSON bytes in a segmented per-collection slab
(fixed 8 MiB segments keep capacity slack under one segment); the docs map
holds flat offsets that stay valid across segment growth, and removed
documents leave garbage bytes until compaction rewrites. The per-document
ArenaAllocator and its second full Pair-tree copy are gone.
The matcher walks the stored bytes directly, skipping by length any field
the filter does not name (a new bson byte-walker: element_key, skip_value,
read_value with borrowed leaves, get_at, and a borrowed spine parse). The
byte matcher is differential-tested against the tree matcher on a corpus
and shares its operator logic. Stored documents are never materialized on
the scan path or in aggregate $match; $group reads group keys and sums
straight off the bytes. Sort, projection, findAndModify, updates and
index entry generation use a borrowed spine into the slab (or the byte
collector, which also replaced collect_values in build_entries). The
compaction threshold now counts uncompressed data volume, since a
compressed log would otherwise never trigger.
Measured (tests/e2e/results/phase5.txt): server RSS 1979 -> 539 MB (2.4x
smaller than MongoDB; phase1 baseline 2.0 GB), range-scan 22.5 -> ~12 ms
(parity, best run faster than MongoDB), proj 4.1 -> 3.4 ms, createIndex
parity. Verified: unit suite in all three modes with zero leaks, the
crash pair, e2e6, and the stress/spill programs.
The log is now a 16-byte file header (magic, version, codec, block
target) plus a sequence of blocks. Each block keeps the pre-existing
record framing unchanged, so Engine.apply_record does not change; records
never straddle blocks (appends accumulate in memory and the block seals
at ~256 KiB). The block header's integrity hash covers the stored payload
bytes exactly as they sit on disk, so the decompressor only ever sees
input already proven intact. Torn tails stay distinguishable from
interior corruption exactly as before: a short read, an impossible
length, or a hash mismatch in the final block truncates cleanly (later
appends overwrite the garbage); a hash mismatch anywhere else is
error.InvalidLog.
The codec is a hand-rolled LZ4 block compressor/decompressor (~1.7 GB/s
measured) with a per-block codec byte falling back to raw when
compression does not help; the header keeps raw legal so zstd can be
swapped in later. Zig 0.16 ships zstd decompression only, and deflate
would cap writes below the insert rate.
Engine.compact goes through the same Log API (deferred sync, one commit)
and compresses for free; sync() seals the pending block before fsyncing,
so the acknowledged-write durability semantics are unchanged (an
unsealed block holds only unacknowledged batch records).
Measured (tests/e2e/results/phase4.txt): db on disk 1025 -> 97 MB, now
smaller than MongoDB's own compressed files; bulk insert 816 -> 722 MB/s
(the accepted compression cost); reopen unchanged at 0.8 s.
Verified: unit suite in all three optimize modes (new LZ4 round-trip,
corrupt-block, and torn-tail truncation tests), the crash pair, e2e6
(kill -9 mid-write), and two full benchmark runs.
Give every Collection an implicit _id_ index (a normal Index with keys
[_id: 1]) so _id equality, $in, ranges and sorts stop depending on the
docs-map hash or a full scan. Kept out of the secondary indexes list, so
listIndexes/dropIndexes/createIndex and the log format are unchanged (no
index_create record, no double listing) and e2e3.js passes unmodified.
Maintained in upsert through the same reserve-then-insert protocol as
the secondaries, removed in evict_doc, and rebuilt after replay by
build_all_indexes alongside them (never maintained mid-replay, so a
failed add can't leave the index under-approximating). index.plan now
takes it as a separate argument. Its keys are canonical
(bson.encode_key gives int32 1, int64 1 and double 1.0 identical bytes),
so the serialization-guarded docs-map fast path (plan_id,
value_fast_path_safe and friends) is deleted.
Measured (tests/e2e/results/phase3.txt): sort({_id:-1}).limit(20) 6.2 ->
2.4 ms (2.3x slower than MongoDB -> parity); integer/string _id point
lookups, $in and ranges verified against the tree. Unit suite in all
three optimize modes, the crash pair, e2e3/e2e4/e2e6.
Replace Index.entries (one sorted array) with a B+tree so writes into an
already-built index stop being quadratic. Nodes are fixed 4 KiB slotted
pages in a flat u32-addressed ArrayListUnmanaged(Node); records longer
than a quarter page spill to an append-only overflow slab (BSON strings
reach 16 MB). Leaves are doubly linked for ordered iteration; the flat
node array stays one contiguous byte range for a later checkpoint.
Insertion descends by separator and splits leaves/internals upward,
promoting keys via a stable copy (a nested split can otherwise clobber
the promoted-key scratch). Deletion does not rebalance: emptied leaves
are unlinked and dropped from their parent, internal nodes may carry one
child, and dead pages are abandoned in place (node memory peaks at the
tree's peak size, exactly what the old array's capacity did). Lookups
are lower-bound seeks plus leaf-chain band scans, so equal keys may
span leaves freely. Bulk build (append_doc_entries + finish_bulk) sorts
a staging array and packs leaves bottom-up. reserve_for now takes the
built entries and reserves exact overflow bytes plus a worst-case node
count, keeping insert_entries infallible after the log append.
db.zig: TTL sweep now seeks the minimum-datetime encoded key and walks
the contiguous datetime band, stopping at the cutoff or type change.
Measured (tests/e2e/results/phase2.txt): updateMany 17.3 -> 1.8 ms
(2.8x slower than MongoDB -> 3.7x faster), createIndex 62 -> 51 ms.
Verified: unit suite ReleaseFast/ReleaseSafe/Debug (incl. the existing
lookup_range and remove_doc differentials, plus a new incremental
insert/remove differential against a brute-force model), the crash pair,
e2e3/e2e4/e2e6, and dev stress tests for depth-2 splits, full drains,
and spilled records through internal levels.
Matching allocated an ArrayList per filter field per candidate document,
on the process-wide allocator, to hold what is almost always a single
value. Candidates now collect into a stack buffer that spills to the heap
only for arrays: measured 15.7 -> 12.0ms on a 65,536-document range scan.
The OOM-propagation test moves with it. Its point is that a failed
collection must surface as an error rather than an empty candidate list,
which would make $ne and $exists:false report a match -- a wrong answer
rather than a failed one. That invariant still holds on the spill path, so
the test now uses an array long enough to reach the allocator, and a new
test pins the flip side: the common single-value match now completes
correctly even when the allocator always fails, because it never calls it.
Query operators were dispatched by a chain of up to fourteen mem.eql per
value per document, with $gt/$gte/$lt/$lte re-comparing the operator name
inside the loop over candidate values. Names resolve to an enum once per
filter field. Command dispatch likewise walked a 30-entry table comparing
strings; it is a comptime StaticStringMap now.
Each request built a fresh reply arena and handed its pages straight back.
One reply per connection, reset between requests, keeps them.
countDocuments() arrives as [{$match: F}?, {$group: {_id: <literal>,
n: {$sum: 1}}}], which the general path answered by materializing every
matching document and discarding them all. It is now recognized and
answered from a counting scan: countDocuments({}) 2.3 -> 1.5ms.
The detector is deliberately conservative -- grouping by "$field", summing
a field, an unmodelled accumulator or any extra stage all fall through to
the general path, since those need the documents themselves. A unit test
pins each accept and reject, and the whole count path was checked against
the general one through the real driver, including the shapes that must
not take it.
The filtered range-scan row does not move: it is bound by walking 65,536
documents that each live in their own arena, not by the matcher. That is
Phase 4 work.
Verified: 77 unit tests under ReleaseFast and ReleaseSafe, e2e
29/16/17/3/2, the crash pair, e2e6 72/72.
Reproduce with bash tests/e2e/compare-run.sh 1g 16k. Without a stored
baseline the phase gates in the plan are not checkable and the projected
numbers are not falsifiable.
Two streams of work land together: they are interleaved in storage.zig
and db.zig and only build as a unit.
Already in the working tree before this session:
- ReleaseFast as the default zig build (Debug was 10-200x slower)
- group commit: one fsync per write command instead of per document
- plan_id returned a pointer to a stack temporary; ReleaseFast read
garbage and silently broke findOne({_id: ObjectId})
- perf suite: big.js, compare.js, compare-run.sh, e2e6.js
Phase 1 performance work:
Record integrity hash CRC32 -> XxHash3. std.hash.Crc32 is the
table-driven byte-at-a-time Crc32IsoHdlc, measured at 408 MB/s against
XxHash3's 31 GB/s: 38us versus 0.5us on a 16 KiB document, which was
about two thirds of the entire bulk-insert cost. The record header
grows from u32 crc to u64 hash (header_len 20 -> 24), a breaking
format change. Bulk insert 260 -> 700 MB/s, reopen 1.1 -> 0.5s.
Compaction fsynced once per live document, because Log.open leaves
defer_sync false and compact never set it. It now issues one sync for
the whole rewrite, before the rename that publishes it.
Compaction triggers on the share of the log that is garbage
(live_docs/dead_docs, maintained at evict_doc, the single point where
a document dies) rather than on bytes appended. A fixed byte count is
wrong in both directions: a 1 GB bulk load holds no garbage at all yet
would compact ~64 times under the 16 MiB default, rewriting 1 GB each
time, while a small collection rewritten in place accumulates garbage
indefinitely without ever reaching the count. Pure inserts now never
compact, and the file stays near 1.25x the live data. Bulk load at the
default threshold: 41.6 -> 702.7 MB/s.
remove() never called maybe_compact, so a delete-heavy workload grew
the log without bound.
e2e6's compaction check required the file to bloat past 30 MiB before
being reclaimed, which encoded the old policy and failed on strictly
better behaviour (ends at 15.8 MB against ~12 MB live, was ~30 MB). It
now asserts the file ends near the live size and peaked well above it,
which does not depend on when the trigger fires. Sampling interval 50
-> 10ms: the operations now finish inside the old window.
Verified: 70 unit tests under both ReleaseFast and ReleaseSafe, e2e
29/16/17/3/2 checks, the crash-a/kill -9/crash-b pair, and e2e6 72/72
across three consecutive runs.
Advertising topologyVersion in the hello reply is what tells a driver the
server speaks the streaming (awaitable) hello protocol — in the Node driver
it is the only condition checked. From the second heartbeat on, the driver
then monitored with an exhaust hello (exhaustAllowed + maxAwaitTimeMS) and
waited for a stream of replies carrying moreToCome. We answered once with
the flag clear and went back to reading, so every heartbeat failed with
"Server ended moreToCome unexpectedly", destroying the connection and
clearing the pool. MongoDB Compass showed this as a connect/disconnect loop
once per heartbeat.
We do not implement streaming hello, so we must not claim to. Omitting the
field keeps monitoring on the polling path, and agrees with the
maxWireVersion 8 we report: streaming hello arrived in wire version 9.
The existing e2e files all passed against the broken server — they issue
their commands and exit before the second heartbeat — so e2e5 watches SDAM
heartbeats on an idle connection instead.
Also renames mongo-light to mongo-lite throughout (binary, log messages,
docs, gitVersion). Unrelated to the fix above, but squashed in at request
rather than left as a commit whose message described only the fix.
createIndex({expireAt: 1}, {expireAfterSeconds: 60}) now deletes a
document once its indexed date is that many seconds old.
index.zig carries the option: Index.ttl (?i64), parsed from
expireAfterSeconds (int32/int64/integral double, within MongoDB's
[0, 2147483647]; 0 means "expire at the stored instant"), emitted by
spec_pairs and so persisted through the log and reported by listIndexes,
and compared by spec_equal — a same-name re-create with a different
expiry stays IndexOptionsConflict, as in MongoDB, since collMod does not
exist here. The bound keeps the value an int32 on the wire and makes the
emission cast unconditional. TTL is single-field only (a compound key is
error.TtlOnCompoundIndex); an absent option parses to null, so every
existing log record reparses unchanged.
db.zig sweeps: Engine.ttl_sweep(now_ms) walks each TTL index's entries
and deletes through the ordinary remove path, so an expiry is logged and
fsynced like any other write and holds across a restart. Expired ids are
duped before removal — remove frees the docs-map key that Entry.id
aliases — then sorted and deduped, because one document can be expired
by several entries (an array of dates expires on its earliest member,
which multikey expansion gives for free) or by several TTL indexes.
Selecting entries is a type test rather than a range lookup: bson
compare order ranks datetime above null, numbers and strings, so a
datetime upper bound would also select every value of a lesser type. A
sweep that deleted something checks the compaction threshold, since a
TTL-only workload never reaches the one in upsert.
commands.zig maps the new spec errors: CannotCreateIndex (67) for a
compound key or an out-of-range expiry, and InvalidIndexSpecificationOption
(197) for an expiry on {_id: 1}, which the idempotent _id no-op would
otherwise swallow.
server.zig runs the monitor as a member of the connection group, so the
existing group.cancel tears it down; it sleeps first, then sweeps under
the write lock, and logs rather than dies on a sweep failure.
--ttl-sweep-secs sets the interval (default 60, 0 leaves it unspawned).
Expiry is coarse by design, as in MongoDB: a document stays visible
until the next sweep, and a non-date value at the indexed path never
expires. Unit tests cover the spec round-trip and every rejection, the
sweep (inclusive cutoff, string/missing/future values untouched, second
sweep a no-op) and its survival of a reopen, and two TTL indexes over
one collection. e2e4.js exercises it through the Node driver.
README documents supported key patterns, unique/sparse/multikey behavior,
planner rules (multikey two-bound range fallback, sparse/null bail, the
_id fast-path guards), and v1 limits plus the two pre-existing issues the
work surfaces (drop-collection resurrection, compact log_bytes).
e2e3.js exercises createIndex/getIndexes/dropIndex/dropIndexes, the
unique-constraint 11000 path, compound, sparse, and descending indexes
through the official Node driver.
The Node driver sends {field: /re/} as a BSON regex element (type 0x0B),
which the matcher previously only handled via the $regex operator form;
and dot paths with numeric segments (tags.0) were ignored because array
descent only recursed into embedded docs. Both are part of standard
MongoDB query semantics and were caught by the driver e2e suite.
server: unbounded Io async limit so the accept loop never wedges, and
treat header-read failures (client RST on pool teardown) as clean
disconnects. With the default cpu_count-1 limit, groupAsync's eager
fallback ran connection handlers inline on the accept-loop fiber once
that many connections were alive, stalling accept() and timing out
handshakes for further clients.
Add tests/e2e/: official driver CRUD, concurrency, and kill -9 recovery
suites (29 + 2 + 3 checks), plus unit tests for the query fixes.
