A whole-index read used to materialize every candidate before the caller saw
the first one. At the tens-of-GB target that is a list of every offset in the
collection -- ~160 MB for a countDocuments({}) over 20 million documents --
which defeats the point of moving storage to disk. Cursors are M1, but
*streaming a scan* has to exist now.
`Candidates` is the one loop candidates arrive through, whatever produced them:
a plan's materialized lookups, or the index read end to end. That keeps this
file's governing invariant -- an index only generates candidates, the full
filter is re-applied to every one -- in a single place. A narrowed plan still
materializes, because its multikey/$in dedupe genuinely needs the whole set and
is bounded by selectivity.
`RevIter` walks `Node.prev`, which has always been maintained and which nothing
had ever read: a descending scan materialized the whole index and reversed the
list. `find({}).sort({_id:-1}).limit(20)` becomes O(20).
The unfiltered fallback now walks the _id_ index instead of the docs map. That
is ordered rather than hash-ordered, and it does not depend on a structure that
is about to be deleted.
`Plan.full_scan()` refuses multikey indexes, since one document contributes
several entries there and a stream cannot dedupe what `search` did. The check is
currently redundant -- the planner refuses to order a multikey index anyway --
and is kept because the two guards protect different things. Stated precisely
in both places after checking: the commands.zig test reddens only when *both*
guards are removed, which is what that test actually pins.
--
This also broke e2e6's compaction check, and the fix there is the more
interesting half.
The check required peak/final > 1.4 and got 1.28. The final size was identical
to the byte (2,398,065 vs 2,398,064) -- compaction reclaimed exactly as before
-- and only the peak moved. Isolated to one variable: changing just the order
updateMany({}) walks its matches moves peak/final between 1.65 and 1.28, because
compaction can also fire from the once-per-second TTL monitor and whether one
lands inside the batch shifts the peak a long way while leaving the outcome
unchanged. The threshold was measuring the schedule.
Replaced with `peak > final`, which measures the shape instead: an append-only
log grows monotonically, so its maximum *is* its final size, and a file that
was ever larger than it ended can only have been rewritten.
Worth recording why the obvious alternative does not work. An absolute size
bound cannot distinguish a working compactor here: the payload is one repeated
character, so ~48 MB of records LZ4-compress to ~3 MB whether or not anything is
reclaimed -- with compaction disabled entirely the file still ends at 3.1 MB. I
first wrote the comment claiming that bound was the strong one, then measured it
and found the opposite; `peak > final` is what goes red.
PLAN amendment A3. The B+tree leaf had nowhere to put a document's slab
offset -- `Slot.extra` is the payload length for a leaf and the child node id
for an internal separator -- which is what blocks the `_id_` tree from becoming
the primary lookup once the docs hashmap goes away.
A leaf record is now `key ++ offset_le`, so `extra` is always 8 and every
byte-accounting site (fits, record_cost, slot_cost, balanced_cut,
repack_keep_prefix) is untouched. Records get *smaller*: an ObjectId `_id_`
record goes from 26 bytes to 21.
`Entry.id` is deleted rather than re-owned. Every entry one document
contributes shares one document, so which document it is belongs on the call
that commits the entries -- which also makes it impossible to confuse the
offset a replace is removing with the one it is inserting. The old field
aliased the docs map's key and was only safe because removal happened at the
one chokepoint where a document dies; that constraint is gone.
Done for secondary indexes too, not just `_id_`. That deletes the per-candidate
`coll.docs.get(id)` in scan_sorted outright rather than replacing it with an
`_id_` descent, and it is free on the write path because a replace already
removes and reinserts every entry in every index.
Consequences worth knowing:
- lookup_eq/lookup_range/Plan.search yield u64. Those are values, immune to the
tree mutation that invalidated the id slices they used to hand back -- which
is why ttl_sweep_coll can drop the dupe-and-free dance it needed to survive
`remove` freeing the key its entries pointed at.
- One safety net is gone. A stale entry used to be swallowed by
`docs.get(id) orelse continue`; now it resolves to superseded-but-parseable
bytes the re-applied filter might accept. That trades an invisible
under-approximation for a visible wrong answer, which is the better failure
to have, but it is a trade.
- A checkpoint may never renumber slab offsets (already recorded in PLAN §4):
every index leaf now holds a physical one.
`zig build fuzz` earned its keep immediately -- it caught the API break in all
four B+tree harnesses, which `zig build test` cannot see.
Benchmarks A/B'd at 256m on one harness, before and after: all rows flat.
updateMany and deleteOne+insertOne first looked 10-13% slower, which three
repeat runs showed to be single-sample noise (0.70/0.71/0.70 against 0.70).
PLAN D2 makes the official specification suites the gate for command semantics;
D7.6 asks for the harness to exist at M0 with a recorded baseline. This is that
harness, pinned on both sides -- mongodb/specifications @ 615e0f9 and
mongodb@7.5.0 -- because a scorecard is only comparable across milestones if a
delta cannot be an upstream test change.
It implements the unified format's Evaluating Matches algorithm as written,
including the two rules that decide whether a pass is earned: extra keys are
tolerated only in a root document, and numeric types compare flexibly. Anything
unimplemented is a SKIP with a reason, never a pass, and the one assertion class
not yet checked -- expectEvents, i.e. command monitoring -- is disclosed at the
top of the scorecard so `pass` reads as an upper bound.
First honest run: 131 pass, 161 fail, 195 skip over 175 files, zero timeouts.
Getting there took four attempts, and the failures are documented in the README
because each would have shipped a scorecard claiming a compatibility gap that
did not exist. Two were genuine leaks in this runner (clients left open when a
case timed out; clients registered for cleanup only after `await connect()`,
plus abandoned cases still creating more). The third I misdiagnosed as machine
load. The fourth attempt found the real cause: a leaked catalog lock in the
engine, fixed separately, which alone accounts for the jump from 45 passes to
131.
So the runner carries its own guards: per-operation CSOT timeouts so work is
never abandoned, an active-handle census per file, an end-of-run tripwire for
stray timers, a hard stop if the server dies rather than emitting hundreds of
misleading ECONNREFUSED failures, and --skip/--limit for bisecting a run whose
failures depend on position. The README states the rule plainly -- a long
unbroken tail of timeouts is a harness bug until proven otherwise -- and the two
commands that settle it.
Also fixes bench-run.sh, which copied its report over bench-latest.txt
unconditionally, including after a run that only warned -- so a degraded run
could silently replace the baseline that PLAN D7.5 makes a milestone gate.
`Message.flags` was parsed and stored but never read. An OP_MSG request with
moreToCome set is fire-and-forget: the client will not read a reply. Sending one
anyway leaves it unread in the socket, so the next command on that connection
reads the previous command's reply and waits forever for its own.
This is not a corner case. Every unacknowledged write uses it, and the Node
driver sends `endSessions` with `writeConcern: {w: 0}` whenever a client closes
-- so an ordinary application that never asks for w:0 still hits it. Before:
insertOne({w: 0}) -> ok, acknowledged=false
countDocuments() (same conn) -> BSON element "cursor" is missing
The command still runs; only the reply is suppressed.
The e2e case pins maxPoolSize to 1, because with a larger pool the driver may
hand the next operation a different connection and hide the bug. It asserts the
connection still works afterwards, which is the part that matters -- not that
the unacknowledged write itself returned.
dispatch resolved the namespace *after* acquiring the catalog lock, and bailed
out with `orelse return` when either part was missing. A plain return is not an
error return, so it ran neither the errdefer nor the explicit unlocks after the
handler: the catalog lock was held, shared, for the life of the process.
`db.aggregate(...)` reaches it. That sends `{aggregate: 1}`, whose value is a
number, so str_arg returns null.
What made this hard to see is that a leaked *shared* lock is invisible to
readers. ping and listDatabases kept answering in microseconds, and the server
looked perfectly healthy from outside -- an external prober got `ok 15ms`
throughout. Only a write needing the catalog exclusive to create a collection
blocked, so the failure surfaced one command later, on a different connection,
as a client-side timeout with nothing to connect it to its cause. It cost three
invalid spec-test baselines before the driver's own command log showed an
insert sitting for exactly socketTimeoutMS against an idle engine.
Namespace resolution now happens before any lock is taken, and a missing name
is a BadValue reply instead of an empty document (which drivers render as the
uninformative "n/a").
Also fixes the aggregate path it exposed: a missing collection returned a reply
with no `ok` field, where MongoDB answers an empty cursor.
Tested by asserting both halves -- a real error reply, and that a following
write which creates a collection still completes. The second is the lock check.
Mutation-checked: reintroducing the leak reddens that test by name.
_id uniqueness was a `coll.docs.contains` probe. The docs hashmap is going
away (PLAN A3), so it has to move to the _id_ tree -- and the tree answers
better, because it is keyed on bson.encode_key, which is canonical where
serialize_value is not. int32 1, int64 1 and double 1.0 are now one _id, as
they are in MongoDB (A4).
_id_ is built and checked first, so a write violating both it and a unique
secondary reports _id_, which is what MongoDB reports. It returns
error.DuplicateKey with `dup_index` left null, which is exactly what
commands.zig's E11000 rendering already treats as "the _id_ index", so the
wire-visible message is unchanged and that file needed no edit.
check_unique's exclude-self became optional and is null on an insert. That was
a latent bug of its own: a replace must ignore its own existing entries, but an
insert has none, and passing the document's id there hides a collision whose
entry carries that same id -- precisely the case _id_ exists to catch. Only
_id_ could reach it, since a secondary collision is between different
documents.
Two corrections found while doing this, both worth reading:
PLAN A4 claimed a database already holding {_id: int32 1} and {_id: int64 1}
loses one on reopen. It does not. Replay evicts through the docs map, keyed on
serialize_value, so both survive; the tree is bulk-built afterwards with
enforcement off, which tolerates duplicate keys and warns. The loss arrives
only with the commit that drops the map, and that is where it needs a
pre-flight scan. Amended.
dispatch_insert asserted only `ok: 1`, but a rejected document comes back as a
writeError alongside it -- so the mixed-type corpus silently shrank from ten
documents to nine when _id_ became unique, and every test over it still passed.
The helper now rejects writeErrors and asserts the inserted count; it caught
the shrink immediately. The corpus keeps an int64 _id on a distinct value, and
the collision it used to stand in for is asserted directly.
Also adds Index.lookup_exact, which the commands that currently probe the docs
map will need. Exact byte equality rather than cmp_prefix, because {a: 1}'s
encoding is a proper prefix of {a: 1, b: 2}'s and a prefix match would claim a
document is present when it is not.
Mutation-checked, all three red: unique=false on id_index; exclude=id_key on
insert; eql -> cmp_prefix in lookup_exact.
Collection.indexes held Index by value, so orderedRemove memmoved the whole
~5 KB struct and every *Index already handed out referred to a different index
afterwards -- a query plan's `index` field, or a slice into an index's
promoted-key buffer. The collection's own bookkeeping stayed consistent, which
is why nothing noticed: only a caller holding a pointer across a drop could
see it, and no test did.
The new test does, and it is mutation-checked against the by-value code that
this commit replaces: holding pointers to b_1 and c_1, then dropping a_1, the
b_1 pointer reads "c_1". Now orderedRemove moves 8-byte pointers, the
surviving indexes do not move, and only the removed one is freed.
M0 needs this independently: an Index will own a file mapping once the node
arena moves into the data file, and copying one by value would duplicate that
ownership.
Not done, though the milestone plan listed it: moving Index's inline scratch
and promo buffers out of the struct. Their stated purpose was to keep those
5 KB out of a file-resident Index and to stop the memmove -- but only the node
arena and overflow slab become file-resident, not the Index metadata, and
boxing already fixed the memmove. Moving them would be churn with nothing left
to buy.
Present since at least d4c9b04, found by the new spec-test harness on its first
run. The $sort stage's materialization branch built its document list with the
reply arena and then handed it to `trees`, whose scope-exit deinit -- and the
$match branch above it -- free with the gpa. So a gpa free was handed an
arena-owned pointer. macOS malloc catches it and aborts with SIGTRAP and no
panic text, which is why the symptom read as "the connection closed":
mfm_free <- Allocator.rawFree
<- array_list.Aligned(*const bson.Document).deinit
<- commands.cmd_aggregate
Any pipeline with $sort and no preceding $group reached it, e.g.
aggregate([{$sort: {x: 1}}]) -- so a client could kill the server with one
ordinary query. With a $group first the stream is already in tree form and the
branch is skipped, which is precisely why it survived: every aggregate case in
e2e.js and e2e6.js sorts *after* grouping.
The list buffer now comes from ctx.gpa. The documents stay in the arena on
purpose -- it outlives the command, and only the ArrayList's own allocator has
to match its deinit.
Tests. The unit test uses a bare $sort pipeline, since a $group first would not
reach the branch, and leans on testing.allocator detecting the invalid free
itself rather than on the host allocator noticing -- mutation-checked by
restoring `arena` on the append, which gives `panic: Invalid free`. The e2e case
adds a second command afterwards, because the assertion that matters is not
that the sort returned rows but that the connection is still there.
Groundwork for M0: the node arena and overflow slab are about to move into an
mmap'd data file where a write to a page belonging to the last durable
checkpoint has to copy that page first (PLAN amendment A1). Two changes make
that a small commit rather than a sixty-site one, plus the reformat of this
file (see the preceding style commit for why it rides along here).
Accessors. Every read of a node page now goes through page(), every write
through page_mut(), and every overflow read through ovf(); nothing else touches
nodes.items or overflow.items. Which of the 55 sites mutate was decided by the
compiler rather than by inspection -- page() returns *const Node, so every
mutating site failed to compile until flipped -- and the result is that the
copy-on-write hook has exactly one home. Records the rule COW will impose
(never hold a *Node across a page_mut of the same id) and the audit showing
today's callers already comply.
Comptime layout asserts. These structures are about to become an on-disk
format, and nothing pinned them. Pinning also surfaced that @sizeOf(Slot) is
32, not the 20 its 160 declared bits suggest -- the backing integer's 16-byte
alignment rounds it up, so 12 of every 32 slot bytes are padding and a node
holds 127 slots where 203 would fit. Pinned, deliberately not fixed: narrowing
the slot changes the fanout and so the on-disk shape of every index, which
belongs in the commit that reshapes leaf records.
reserve_for. The old bound stood in for "levels a batch can add" with n/8,
which is ~125 levels for a 1000-entry batch and demands ~528 MiB of headroom.
Growing by g levels needs at least 2^g entries, so log2_ceil(n+1)+1 bounds it,
giving ~70 MiB for that batch. Harmless as ArrayList capacity; real file growth
once the arena is file-backed. Overrunning the reservation is a buffer overrun
on a path that has already appended to the log and cannot report failure, so
alloc_node and store_record now assert, using assert.zig so the checks survive
ReleaseFast. Mutation-checked by dropping the reservation entirely: six tests
go red with the new message. Worth noting the assert guards the allocation, not
the arithmetic -- ensureUnusedCapacity over-allocates, so a slightly-too-small
bound is masked until the reservation becomes exact.
build.zig gains a `fuzz` step. spill, spill2, stress and fuzz_split were in no
build step and are not in lib.zig's test block, so `zig build test` could not
see an API break in the only coverage for records past the inline limit and for
randomized split/remove interleavings -- exactly what this work puts at risk.
The M0 implementation review found three of D1-D9 wrong or incomplete. The
originals stay in place with pointers to a new amendments section, so a later
session can see what changed rather than reading a rewritten history.
A1: D4 as written is unsound. The doc and overflow slabs are append-only, so
replay repairs them, but B+tree node pages are mutated in place -- after a
crash the file holds an arbitrary mix of written-back and not-written-back
pages, and once D6.3 truncates the log the data file is the only copy below
the watermark. A half-persisted tree is unrecoverable. So the checkpoint needs
shadow paging: no page below the last watermark's allocation mark is ever
stored into, and the watermark write is the atomic switch. Knock-on: node ids
cannot be page numbers, because Node.parent/next/prev are back-pointers by id
and copy-on-write would cascade; an in-RAM id->page table per index keeps every
persisted id at its current width and gives COW one pointer to fix.
A2: follows from A1 -- a page free list is a prerequisite, not the
defense-in-depth D6.2 assumed, because COW abandons every page it touches in
every epoch. The churn gate stays, retargeted at document garbage.
A3: section 5 step 4's trap was misidentified. store_record already copies
keys, so "entries must own their key bytes" is work that does not need doing.
The real problem is that a leaf record has nowhere to put a slab offset; the
resolution is to make the payload that offset, in every index, and delete
Entry.id rather than re-own it.
A4: making _id_ a unique index keys uniqueness on the canonical encode_key
rather than serialize_value, so int32 1 / int64 1 / double 1.0 collide as they
do in MongoDB -- a compatibility improvement, with a documented one-way
migration hazard for a database that already holds two such documents.
Also records that Engine.seq is never restored on open (harmless today, silent
data loss once a watermark exists), and two bugs the new spec harness found.
AGENTS.md carries the same rules into the operating guide: ground rules grow
from 7 to 9, and the old rule 6 is corrected with a note saying why.
Wrap signatures and long expressions to the 100-column limit and make every
file zig fmt clean. Semantics-preserving throughout: ignoring whitespace and
the trailing commas that wrapping introduces, every file here is byte-identical
to its predecessor, and the one apparent exception is a warning string split
with `++`, which concatenates at comptime to the same bytes.
src/index.zig and src/commands.zig are reformatted in the commits that follow,
because their reformat is interleaved with in-flight changes to them and
separating the two would need the reformat re-derived rather than moved.
Prose and benchmark tables use MultiforaDB; the binary, the CLI usage
line, the log-message prefix and the default database file use
multiforadb.
Two consequences worth noting:
- build.zig.zon's fingerprint is derived from the package name, so it
had to change with it (Zig refuses to build otherwise). A consumer
pinning this package by fingerprint needs updating.
- the default --db path is now multiforadb.log, and getCmdLineOpts
reports it as dbpath. An existing mongo-lite.log has to be passed
explicitly with --db.
The e2e harness abbreviated the old name as ML_; that is now MFDB_,
including the documented ML_BIN override (MFDB_BIN) and the scratch
file names. MD_ (mongod) is untouched.
compare-run.sh spawned the server by absolute path under a
sandbox/mongo-lite directory that no longer exists; that block already
runs from tests/e2e, so it uses a relative path now.
The archived reports under tests/e2e/results/ keep the old name: they
record what the old binary measured.
compare-run.sh answers "how do we compare to MongoDB"; it says nothing
about whether a change made things better or worse than last week. Add a
harness that records each run and diffs it against the previous one.
bench-run.sh wraps compare-run.sh, adds a concurrent durable-write
comparison (concurrent.js: N clients each doing sequential insertOne with
{w:1, j:true}, exercising the group-commit path under real contention),
writes a versioned name<TAB>value report to results/bench-<timestamp>.txt,
and prints a diff of our numbers against results/bench-latest.txt.
Also:
- compare-run.sh polled with fixed sleeps, which are flaky once earlier
benchmark phases have warmed the machine; both servers now wait on a
real driver connection instead.
- a dispatch error only reached the client as a generic InternalError,
with nothing on the server side naming the failing command; log the
connection, command and error name before replacing the reply.
Follow-up hardening on the group-commit work from ecd28d9 and c8d547f. The
signal -> broadcast fix and the append-drained wakeup under commit_lock were
correct but incomplete; each of the three defects below could lose or corrupt
data that had already been acknowledged.
- log_append's cleanup defer took commit_lock with `lock(...) catch {}` and
then unlocked unconditionally. Mutex.lock is Cancelable!void and
Mutex.unlock treats an already-unlocked mutex as `unreachable`, so a
cancellation there (client disconnect, shutdown) released a mutex the fiber
never held: a panic in ReleaseSafe and silent memory corruption in the
default ReleaseFast build. A cleanup path must not be a cancellation point,
so it uses lockUncancelable.
- Engine.commit waited with `catch return`, which returns *success* from
!void. A failed wait therefore told the caller its write was on disk and the
dispatch epilogue replied ok without a seal or an fsync -- the same failure
class c8d547f fixed, reached through the error path instead of the happy
one. The follower wait now propagates; the leader's drain is uncancelable,
since once `committing` is set every other writer is parked behind it and
the drain is bounded anyway.
- Two compactions could run at once. They share one `<log>.tmp` path and each
ends in a rename onto the log, so one truncates and rewrites the file the
other is about to publish, and then that one renames whatever it finds over
the live log. compact now claims an atomic `compacting` slot and a second
caller returns; the guard sits on the resource rather than in take_compact,
so direct callers (tests included) are covered too. compact_pending became
atomic while we were there: note_compact sets it under a *collection* lock
and the epilogue read it under none.
Also in compaction: the tmp file is opened with a new Log.create that
truncates. Log.open keeps an existing file's bytes and only rewinds end_pos to
the header, so a longer tmp left by a crashed or retried rewrite kept its
tail -- and those trailing blocks are intact and hash-correct, so replay
applied them as live records once the rename published the file, resurrecting
deleted documents. The `deleteFile ... catch {}` that used to stand in for
this is gone, and with it a swallowed error the invariant rested on. The
retry loop is bounded at 8 attempts (each one rewrites the whole log before
the seq check can reject it, so an unbounded retry livelocks under sustained
writes); giving up re-arms the request instead of failing the write.
A compaction failure no longer fails the write whose epilogue triggered it:
the write is durable by then, so the error is reported and the request
re-armed rather than turned into an error the client retries.
Assertions: db.zig, commands.zig and storage.zig had none, which is why every
defect in this series was found by a stress run rather than at the moment of
corruption. src/assert.zig adds an assert that survives ReleaseFast --
std.debug.assert lowers to `unreachable`, which in this project's default
build is not a skipped check but a promise to the optimizer, exactly the wrong
lowering for a durability invariant that might be false. Eleven of them now
cover the commit watermark, the in-flight append count, the compaction
snapshot, and the live/dead counters (whose u64 subtraction would otherwise
underflow into a live count that suppresses compaction forever).
cmd_find asserts that a missing collection implies no matches instead of
silently emitting an empty page for a query that did match.
Dead state removed: Log.defer_sync was never read (only written once by
compact), yet its doc comment instructed callers to follow a defer_sync
protocol that no longer exists and has no effect if followed. Engine's
begin_batch/end_batch were both `_ = self`, so three call sites announced a
batch boundary that wasn't there. Both are gone and the comments now describe
the real contract: appends never sync, Log.sync is the only commit point.
Tests: Log.create's truncation is pinned by a storage test that replays after
reusing a path, and the compaction guard by a db test that drives the flag
directly -- both confirmed to fail without their fix. The threaded test added
alongside them is a smoke test only, and says so: both races have windows too
narrow to hit reliably (compact_snapshot_coll holds each collection's write
lock while snapshotting, so two compactions serialize there and an insert
cannot re-arm compact_pending meanwhile), and it passes with the guard
removed.
Includes an unrelated fix that was already in the working tree: slab_append
held a pointer into slab.items across an append to that same list, which
could dangle after a realloc and corrupt the new segment's start offset.
Verified: unit suite in ReleaseFast/ReleaseSafe/Debug; e2e, e2e2 concurrent,
e2e3, e2e4, e2e5, e2e6 (72/72) and the kill -9 crash pair, with no assertion
firing anywhere.
Three defects, each of which made the database lose data that had already
been acknowledged, or answer a client with a malformed reply.
- Engine.commit decided a writer was "already covered" by comparing
log.end_pos with the position of the last completed commit. Under block
framing an append leaves its bytes in the log's open in-memory block and
does not move end_pos -- only sealing does. So once the first commit had
set committed_end = end_pos, every later write command found itself
covered and returned without sealing or syncing anything. A no-op
deleteMany followed by insertMany(50) was acknowledged with the file
still 16 bytes (its header) and lost all 50 documents on kill -9, which
is precisely what e2e2's crash pair does. Coverage is now decided by
sequence number, which counts records rather than bytes on disk.
- Compaction read the new log's end position before syncing it, but the
sync is what seals the open block, and the seal is what moves end_pos
past it. Appends after a compaction therefore started inside the
compacted file's last block and overwrote it, so those documents were
gone at the next replay: e2e6's phase 2 ended with 1000 documents in
memory and 996 after a graceful restart.
- cmd_find returned early on a missing namespace without putting anything
in the reply, so a find on an unknown collection arrived at the driver as
a response with no `ok` field ("MongoServerError: n/a") instead of an
empty cursor. The other commands' missing-namespace paths were fine.
Verified with the unit suite in ReleaseFast/ReleaseSafe/Debug, the split
fuzzer, all six e2e suites (e2e6 back to 72/72) and the kill -9 crash pair
-- none of which passed beforehand -- plus 13 kill -9 runs over 1/2/8
connections with 1200 acknowledged inserts each and nothing lost.
tests/e2e/results/phase7.txt records the benchmark with the fixes in place:
no regression against phase6 (bulk 739 -> 753 MB/s, updateMany 1.9 -> 2.0
ms, RSS 547 -> 546 MB), and concurrent durable writes now measurable at
7.1k/15.0k/21.8k docs/s over 1/8/32 connections.
The single engine-wide reader/writer lock is replaced by a lock hierarchy,
so writes to different collections no longer serialize on one mutex:
- Collections are heap-allocated, so their addresses are stable while a
command holds a collection lock (the maps only store pointers).
- A catalog rwlock guards the database/collection maps: shared for every
command (so a concurrent DDL cannot mutate the maps underneath it),
exclusive for create/drop/dropDatabase. Each collection has its own
rwlock; the ordering is always catalog -> collection -> log lock, never
two collection locks at once (TTL sweep and compaction take collections
one at a time).
- Command dispatch acquires the catalog + target collection locks for the
handler's duration, resolving the collection (creating it for writes)
under the catalog lock; create/drop upgrade to the exclusive catalog lock.
- Appends never fsync. Each write command's epilogue releases the
collection lock, then commits once (seal + fsync) with a leader/follower
group commit: the leader waits for writers mid-append (a pending counter)
so its seal covers them, and followers whose records the seal covered
skip their own fsync. Every acknowledged write is fsynced before its
reply (crash pair verified); an unacknowledged write may vanish and a
reader may observe a write before its fsync — ordinary w:1 j:true
semantics instead of 'the log describes >= memory'.
- Compaction snapshots collections without the log lock (so a concurrent
writer holding one can always finish its append) and retries when a
writer appended mid-snapshot (detected via the record seq), then swaps
under the log lock — no deadlock. The compaction trigger moved to the
command epilogue and the TTL monitor.
- Engine.dup_index moved to the collection (per-command error paths).
Also lands two B-tree edge-case fixes driven by tests that were in flight:
a churned leaf full of dead bytes no longer splits with an empty right
half (the leaf is repacked before splitting, and an emptied node's page is
fully free again), and a slot-count split with all large records on one
side shifts records between the halves until the new record fits. Plus a
randomised fuzz test over key sizes (src/fuzz_split.zig) and the two
regression tests.
Measured (tests/e2e/results/phase6.txt): no regression on the
single-connection benchmark; concurrent durable-insert throughput ~5.1k ->
12.5k docs/s from 1 -> 8 clients, ~14.8k at 32. Verified: unit suite in
all three modes, all e2e suites, the kill -9 crash pair.
Documents live as canonical BSON bytes in a segmented per-collection slab
(fixed 8 MiB segments keep capacity slack under one segment); the docs map
holds flat offsets that stay valid across segment growth, and removed
documents leave garbage bytes until compaction rewrites. The per-document
ArenaAllocator and its second full Pair-tree copy are gone.
The matcher walks the stored bytes directly, skipping by length any field
the filter does not name (a new bson byte-walker: element_key, skip_value,
read_value with borrowed leaves, get_at, and a borrowed spine parse). The
byte matcher is differential-tested against the tree matcher on a corpus
and shares its operator logic. Stored documents are never materialized on
the scan path or in aggregate $match; $group reads group keys and sums
straight off the bytes. Sort, projection, findAndModify, updates and
index entry generation use a borrowed spine into the slab (or the byte
collector, which also replaced collect_values in build_entries). The
compaction threshold now counts uncompressed data volume, since a
compressed log would otherwise never trigger.
Measured (tests/e2e/results/phase5.txt): server RSS 1979 -> 539 MB (2.4x
smaller than MongoDB; phase1 baseline 2.0 GB), range-scan 22.5 -> ~12 ms
(parity, best run faster than MongoDB), proj 4.1 -> 3.4 ms, createIndex
parity. Verified: unit suite in all three modes with zero leaks, the
crash pair, e2e6, and the stress/spill programs.
The log is now a 16-byte file header (magic, version, codec, block
target) plus a sequence of blocks. Each block keeps the pre-existing
record framing unchanged, so Engine.apply_record does not change; records
never straddle blocks (appends accumulate in memory and the block seals
at ~256 KiB). The block header's integrity hash covers the stored payload
bytes exactly as they sit on disk, so the decompressor only ever sees
input already proven intact. Torn tails stay distinguishable from
interior corruption exactly as before: a short read, an impossible
length, or a hash mismatch in the final block truncates cleanly (later
appends overwrite the garbage); a hash mismatch anywhere else is
error.InvalidLog.
The codec is a hand-rolled LZ4 block compressor/decompressor (~1.7 GB/s
measured) with a per-block codec byte falling back to raw when
compression does not help; the header keeps raw legal so zstd can be
swapped in later. Zig 0.16 ships zstd decompression only, and deflate
would cap writes below the insert rate.
Engine.compact goes through the same Log API (deferred sync, one commit)
and compresses for free; sync() seals the pending block before fsyncing,
so the acknowledged-write durability semantics are unchanged (an
unsealed block holds only unacknowledged batch records).
Measured (tests/e2e/results/phase4.txt): db on disk 1025 -> 97 MB, now
smaller than MongoDB's own compressed files; bulk insert 816 -> 722 MB/s
(the accepted compression cost); reopen unchanged at 0.8 s.
Verified: unit suite in all three optimize modes (new LZ4 round-trip,
corrupt-block, and torn-tail truncation tests), the crash pair, e2e6
(kill -9 mid-write), and two full benchmark runs.
Give every Collection an implicit _id_ index (a normal Index with keys
[_id: 1]) so _id equality, $in, ranges and sorts stop depending on the
docs-map hash or a full scan. Kept out of the secondary indexes list, so
listIndexes/dropIndexes/createIndex and the log format are unchanged (no
index_create record, no double listing) and e2e3.js passes unmodified.
Maintained in upsert through the same reserve-then-insert protocol as
the secondaries, removed in evict_doc, and rebuilt after replay by
build_all_indexes alongside them (never maintained mid-replay, so a
failed add can't leave the index under-approximating). index.plan now
takes it as a separate argument. Its keys are canonical
(bson.encode_key gives int32 1, int64 1 and double 1.0 identical bytes),
so the serialization-guarded docs-map fast path (plan_id,
value_fast_path_safe and friends) is deleted.
Measured (tests/e2e/results/phase3.txt): sort({_id:-1}).limit(20) 6.2 ->
2.4 ms (2.3x slower than MongoDB -> parity); integer/string _id point
lookups, $in and ranges verified against the tree. Unit suite in all
three optimize modes, the crash pair, e2e3/e2e4/e2e6.
Replace Index.entries (one sorted array) with a B+tree so writes into an
already-built index stop being quadratic. Nodes are fixed 4 KiB slotted
pages in a flat u32-addressed ArrayListUnmanaged(Node); records longer
than a quarter page spill to an append-only overflow slab (BSON strings
reach 16 MB). Leaves are doubly linked for ordered iteration; the flat
node array stays one contiguous byte range for a later checkpoint.
Insertion descends by separator and splits leaves/internals upward,
promoting keys via a stable copy (a nested split can otherwise clobber
the promoted-key scratch). Deletion does not rebalance: emptied leaves
are unlinked and dropped from their parent, internal nodes may carry one
child, and dead pages are abandoned in place (node memory peaks at the
tree's peak size, exactly what the old array's capacity did). Lookups
are lower-bound seeks plus leaf-chain band scans, so equal keys may
span leaves freely. Bulk build (append_doc_entries + finish_bulk) sorts
a staging array and packs leaves bottom-up. reserve_for now takes the
built entries and reserves exact overflow bytes plus a worst-case node
count, keeping insert_entries infallible after the log append.
db.zig: TTL sweep now seeks the minimum-datetime encoded key and walks
the contiguous datetime band, stopping at the cutoff or type change.
Measured (tests/e2e/results/phase2.txt): updateMany 17.3 -> 1.8 ms
(2.8x slower than MongoDB -> 3.7x faster), createIndex 62 -> 51 ms.
Verified: unit suite ReleaseFast/ReleaseSafe/Debug (incl. the existing
lookup_range and remove_doc differentials, plus a new incremental
insert/remove differential against a brute-force model), the crash pair,
e2e3/e2e4/e2e6, and dev stress tests for depth-2 splits, full drains,
and spilled records through internal levels.
Five items in dependency order, each sized to land on its own. The design
decisions already settled are recorded so they are not re-derived, and so
are the ordering constraints, which are the part that actually matters --
notably that the _id index must follow the tree, because it updates on
every insert and against a sorted array that is only affordable while ids
happen to append at the end.
Also carries the ground rules the earlier work established: A/B on one
harness rather than trusting a model, mutation-check tests that guard an
invariant, and the two absolutes (an index may only over-approximate; the
database must always open).
Each item names the traps found while investigating it -- deletion being
where B+trees go wrong, listIndexes possibly noticing a real _id_ index,
hashing compressed rather than uncompressed bytes, the fabricated
Document values that break when Document changes meaning, and the
durability guarantee that quietly weakens under group commit.
The performance table, the large-collection notes and the roadmap all
described the state before the optimization work.
The table is the recorded Phase 1 gate run (tests/e2e/results/phase1.txt).
Bulk insert, createIndex and reopen moved from losses to wins; the sort
row is split, since an indexed field now streams out of the index at 1.0ms
while _id still materializes.
The large-collection notes carried advice that is now wrong: compaction no
longer needs a raised --compact-threshold for bulk loads, and building an
index over existing data is no longer quadratic (inserting into one that
already exists still is). The v1 index limits no longer claim removal is
O(n) or that sort cannot use an index.
The roadmap becomes what is left, in order, with the reason each remaining
gap is structural, plus a record of what was done and the measurement that
drove it. Two more bugs join the list: compaction fsynced once per live
document, and remove never checked the compaction threshold.
A sort ordered every match before discarding all but one page, even when
an index already held the candidates in exactly that order. The planner
now recognizes that case and the scan streams the page straight out.
An index provides the sort when the sort keys line up with the components
after the equality-pinned prefix (those are fixed to one value each, so
they do not affect the order of what follows) and every direction agrees
uniformly -- all the same way round, or all opposite, since the array can
only be read forwards or backwards. Multikey indexes are excluded: they
emit a document once per indexed value, so their order is not an order on
documents. So is an $in, whose disjoint ranges concatenate unordered.
Entries already come out of the array in key order, so forward scans were
sorted all along; what destroyed it was the dedupe pass sorting by id.
The conditions above are exactly the ones under which that pass is
skipped, so ordered output needs only reversing for a backward scan.
evaluate_index gains a plan shape it did not have: a full index scan when
the ordering is the reason to use it. Without that, find({}).sort(...)
was unreachable -- an empty filter yields no clauses and the planner bailed
before looking at any index. It is guarded to only appear when the sort is
satisfied, since otherwise scanning the docs map directly is cheaper.
The early stop had to move into the scan, which is the only place that
knows whether the order came from an index: a limit is a valid page
boundary without a sort, or with one an index supplies, and otherwise
means nothing. Getting this wrong the other way -- limiting first and
re-scanning -- would have doubled the work for every unindexed sort.
find({}).sort({k: 1}).limit(20) over 65,536 x 16 KB documents:
4.0ms -> 1.0ms
sort({_id: -1}) is unchanged at 4.3ms: nothing ordered covers _id yet.
Checked against the definition rather than by example: eleven query shapes
-- forward, backward, skip, unlimited, filtered, equality-prefixed both
directions, compound, directions the index cannot serve, and a range --
each compared through the real driver against the page of the equivalent
full sort.
Verified: 78 unit tests under ReleaseFast and ReleaseSafe, e2e
29/16/17/3/2, the crash pair, e2e6 72/72.
remove_id walks every entry in the index comparing ids, so evicting one
document cost O(index size) per index -- on a 65,536-document collection
that is 65,536 id comparisons to remove a single entry, and it ran on
every update and every delete.
Entry generation is a pure function of the document, so remove_doc
regenerates the entries the document contributed and binary-searches for
each one. evict_doc now removes the document from the docs map first and
hands the document itself to the index, while both it and the map key are
still alive.
updateMany({k: 7}, {$inc}) over 65,536 documents, measured A/B:
15.4ms -> 5.5ms (MongoDB 8.3.7: 6.1ms)
It is infallible by construction. Regeneration allocates and can fail,
and a document the index could not key contributed nothing to remove; in
either case it falls back to the scan, which is always correct. It also
falls back if the regenerated entries are not all found, so a
disagreement degrades to slow rather than leaving a stale index. That
last guard is defensive only -- regeneration is deterministic, so the
test below does not reach it.
The equivalence is checked directly rather than by example: two identical
indexes are built over documents exercising multikey arrays with repeats,
missing fields and both sparse settings, then emptied document by
document in shuffled order -- one through remove_doc, one through
remove_id -- asserting the entry arrays stay byte-identical at every
step. Verified it fails when removal order is reversed, which is the way
positional removal actually breaks.
Insertion still memmoves the tail. That, and ordered leaf iteration, are
what the tree is still for.
Note for later: the updateOne({_id}) and deleteOne({_id}) paths are slow
here for an unrelated reason -- an integer _id is rejected by
value_fast_path_safe, so each one is a full collection scan. The encoded
keys already make that guard unnecessary; removing it belongs with the
_id index.
Verified: 78 unit tests under ReleaseFast and ReleaseSafe, e2e
29/16/17/3/2, the crash pair, e2e6 72/72.
Entry.key becomes the order-preserving byte encoding of the indexed
values, concatenated column by column, instead of a slice of Values.
Comparing two entries is now a memcmp.
The old representation allocated one Value slice per entry, and each
Value in it pointed into a different document's arena -- so a binary
search over the entry array was a chain of pointer chases across the
heap, and every comparison walked the key component by component
dispatching on BSON type. Byte keys make the comparison contiguous and
type-free, and the key no longer aliases the document at all.
createIndex over 65,536 documents:
{k: 1} 56ms -> 44ms
{s: 1} unique 53ms -> 31ms
{p: 1, k: -1} 54ms -> 37ms
(both already down from ~650ms before the bulk build)
The search API still takes Values and encodes at the call site: lookups
happen per query, not per document, so there is nothing to gain from
pushing the encoding out to callers, and Plan keeps its current shape.
Prefix search compares raw byte prefixes, which is sound because every
column encoding is self-delimiting -- a prefix of an encoded key is
exactly the encoding of its leading columns. For the same reason a
complete column encoding can never be a proper prefix of another, so
finish_bulk's duplicate test is now a plain byte equality.
The TTL sweep read entry keys as Values to find datetimes. It now uses
bson.encoded_leading_datetime, which checks the column's tag and decodes
eight bytes rather than the whole key. Still a linear walk for the reason
the existing comment gives.
Key direction is deliberately still not applied to the encoding.
Complementing descending columns would let a sort read the array
forwards, but nothing exploits that yet, and doing it now would change
the array's order for no gain. It belongs with the sort-aware planner.
remove_id is still a linear scan and insertion still memmoves the tail:
those are the tree's job, not this change's.
Verified: 77 unit tests under ReleaseFast and ReleaseSafe, e2e
29/16/17/3/2, the crash pair, e2e6 72/72, and the randomized
lookup_range test that checks bounds against a brute-force filter.
Encodes a Value so that std.mem.order over the bytes reproduces
bson.compare exactly. This is the foundation for the encoded-key index:
it lets an index binary-search, range-scan and eventually be stored as
raw bytes, instead of carrying Value trees whose every comparison chases
pointers into a different document's arena.
Layout is [rank + 1] then a self-delimiting payload; the +1 keeps 0x00
out of the tag space so it can terminate variable-length payloads.
The parts that are easy to get wrong, and why they are the way they are:
Numbers encode the f128 that compare already widens int32, int64 and
double to -- exactly, for all three. So int32 1, int64 1 and double 1.0
produce identical bytes, which is the cross-type equality that numeric
index lookups need, and is precisely what value_fast_path_safe exists
today to work around. Negatives are bit-inverted and positives get the
sign bit set, making the IEEE order lexicographic. -0.0 normalizes to
+0.0 (they compare equal) and every NaN encodes as all-ones (compare
makes NaN greatest and all NaNs equal).
Byte strings escape 0x00 as 00 FF and terminate with 00 00. A BSON
string may contain NUL, so a bare terminator would be ambiguous;
escaping fixes ordering at the same time, since a real NUL then sorts
above the terminator and any byte >= 01 does too. "Shorter is less"
falls out to match std.mem.order, which also gives documents and arrays
their length tie-break for free.
Binary length-prefixes because compare_binary orders by length first,
but opaque_val escapes instead: compare ignores its kind and orders the
data lexicographically, not by length.
Correctness rests entirely on the order equivalence, so it is checked
exhaustively rather than by example: every ordered pair of a corpus
spanning all fifteen ranks and their boundaries (numeric cross-type and
sign, NaN, both zeros, infinities, embedded NULs, prefix relationships,
empty and nested documents and arrays, binary subtypes) is compared both
ways. A second test concatenates two-column keys and checks they
reproduce component-wise order, which is what makes compound keys and
prefix search sound. Verified both fail when escaping is dropped, when
-0.0 is not normalized, when the binary length prefix is wrong, and when
NaN stops being greatest.
Nothing uses the encoding yet; the index still holds Value keys.
Matching allocated an ArrayList per filter field per candidate document,
on the process-wide allocator, to hold what is almost always a single
value. Candidates now collect into a stack buffer that spills to the heap
only for arrays: measured 15.7 -> 12.0ms on a 65,536-document range scan.
The OOM-propagation test moves with it. Its point is that a failed
collection must surface as an error rather than an empty candidate list,
which would make $ne and $exists:false report a match -- a wrong answer
rather than a failed one. That invariant still holds on the spill path, so
the test now uses an array long enough to reach the allocator, and a new
test pins the flip side: the common single-value match now completes
correctly even when the allocator always fails, because it never calls it.
Query operators were dispatched by a chain of up to fourteen mem.eql per
value per document, with $gt/$gte/$lt/$lte re-comparing the operator name
inside the loop over candidate values. Names resolve to an enum once per
filter field. Command dispatch likewise walked a 30-entry table comparing
strings; it is a comptime StaticStringMap now.
Each request built a fresh reply arena and handed its pages straight back.
One reply per connection, reset between requests, keeps them.
countDocuments() arrives as [{$match: F}?, {$group: {_id: <literal>,
n: {$sum: 1}}}], which the general path answered by materializing every
matching document and discarding them all. It is now recognized and
answered from a counting scan: countDocuments({}) 2.3 -> 1.5ms.
The detector is deliberately conservative -- grouping by "$field", summing
a field, an unmodelled accumulator or any extra stage all fall through to
the general path, since those need the documents themselves. A unit test
pins each accept and reject, and the whole count path was checked against
the general one through the real driver, including the shapes that must
not take it.
The filtered range-scan row does not move: it is bound by walking 65,536
documents that each live in their own arena, not by the matcher. That is
Phase 4 work.
Verified: 77 unit tests under ReleaseFast and ReleaseSafe, e2e
29/16/17/3/2, the crash pair, e2e6 72/72.
Reproduce with bash tests/e2e/compare-run.sh 1g 16k. Without a stored
baseline the phase gates in the plan are not checkable and the projected
numbers are not falsifiable.
sort+limit ordered the entire result set to return one page: 65,536
documents sorted to hand back 20. Two independent costs.
The decorate pass built, per document per sort key, an ArrayList of every
value at the path -- but the comparator only ever reads element 0. Added
first_value_at, which mirrors collect_values' traversal exactly (same
order, same depth cutoff) and stops at the first hit, and moved the
decorated values into one flat allocation. That equivalence is the whole
correctness argument, so it is pinned by a test covering dotted paths,
arrays of documents, numeric element addressing, repeated keys, missing
paths and the depth cutoff.
sort_docs_top_k keeps a k-element max-heap instead of ordering
everything: one comparison against the heap root per document, and only
the survivors are ever sorted. cmd_find uses it when the page is at most
a quarter of the matches, where the heap's bookkeeping still pays for
itself, and falls back to a full sort otherwise. It leaves docs[k..]
unordered, which is safe because the page is a prefix of the first k.
cmd_aggregate's $sort is deliberately untouched: a later stage can read
the whole stream, and top-k would silently corrupt the tail.
find({}).sort({_id:-1}).limit(20) over 65,536 x 16 KB documents:
baseline 40.0ms
decorate only (top-k disabled) 23.9ms
decorate + top-k 4.3ms
Correctness checked end to end as well: the limited page is identical to
the prefix of the equivalent full sort. The top-k test compares against a
full sort across ascending, descending and compound keys, for k of 1, 2,
20, n-1, n and n+1, over data with heavy ties; verified it fails when the
heap's child comparison is inverted.
createIndex built the entry array one document at a time, and each
insert kept the array sorted by memmoving the tail -- O(n^2) bytes moved
over a full build, which was the entire cost of the operation. Entries
are now appended unsorted and ordered once (append_doc_entries +
finish_bulk), with uniqueness checked by a single adjacent-pair scan
instead of a binary search per document. build_all_indexes, which runs
for every index on every open, takes the same path.
createIndex over 65,536 documents, measured A/B:
{k: 1} 649ms -> 56ms
{s: 1} unique 678ms -> 53ms
{p: 1, k: -1} 653ms -> 54ms
lookup_range binary-searched only the equality prefix and then scanned
that whole band applying a filter, so a range on the first component of
an index touched every entry in it. Both ends are now binary searches
over the component the array is already sorted on, clamped into the
equality band. Note this does not move the range-scan row in compare.js:
that query filters on p, which has no index there, so it is a collection
scan and belongs to the matcher.
cmd_find passed a hardcoded 0 as the scan limit, so find().limit(n)
materialized the entire collection before slicing. It now stops once the
page is filled, when there is no sort to order the matches first; the
bound covers the skipped prefix because the scan counts matches rather
than returned documents.
lookup_range's bounds are checked by a new randomized test that compares
the result count against a brute-force filter over 600 generated
queries, with values chosen from a small domain so equal keys and the
inclusive/exclusive edges come up constantly. Verified it fails when
either bound is swapped.
Two streams of work land together: they are interleaved in storage.zig
and db.zig and only build as a unit.
Already in the working tree before this session:
- ReleaseFast as the default zig build (Debug was 10-200x slower)
- group commit: one fsync per write command instead of per document
- plan_id returned a pointer to a stack temporary; ReleaseFast read
garbage and silently broke findOne({_id: ObjectId})
- perf suite: big.js, compare.js, compare-run.sh, e2e6.js
Phase 1 performance work:
Record integrity hash CRC32 -> XxHash3. std.hash.Crc32 is the
table-driven byte-at-a-time Crc32IsoHdlc, measured at 408 MB/s against
XxHash3's 31 GB/s: 38us versus 0.5us on a 16 KiB document, which was
about two thirds of the entire bulk-insert cost. The record header
grows from u32 crc to u64 hash (header_len 20 -> 24), a breaking
format change. Bulk insert 260 -> 700 MB/s, reopen 1.1 -> 0.5s.
Compaction fsynced once per live document, because Log.open leaves
defer_sync false and compact never set it. It now issues one sync for
the whole rewrite, before the rename that publishes it.
Compaction triggers on the share of the log that is garbage
(live_docs/dead_docs, maintained at evict_doc, the single point where
a document dies) rather than on bytes appended. A fixed byte count is
wrong in both directions: a 1 GB bulk load holds no garbage at all yet
would compact ~64 times under the 16 MiB default, rewriting 1 GB each
time, while a small collection rewritten in place accumulates garbage
indefinitely without ever reaching the count. Pure inserts now never
compact, and the file stays near 1.25x the live data. Bulk load at the
default threshold: 41.6 -> 702.7 MB/s.
remove() never called maybe_compact, so a delete-heavy workload grew
the log without bound.
e2e6's compaction check required the file to bloat past 30 MiB before
being reclaimed, which encoded the old policy and failed on strictly
better behaviour (ends at 15.8 MB against ~12 MB live, was ~30 MB). It
now asserts the file ends near the live size and peaked well above it,
which does not depend on when the trigger fires. Sampling interval 50
-> 10ms: the operations now finish inside the old window.
Verified: 70 unit tests under both ReleaseFast and ReleaseSafe, e2e
29/16/17/3/2 checks, the crash-a/kill -9/crash-b pair, and e2e6 72/72
across three consecutive runs.
Advertising topologyVersion in the hello reply is what tells a driver the
server speaks the streaming (awaitable) hello protocol — in the Node driver
it is the only condition checked. From the second heartbeat on, the driver
then monitored with an exhaust hello (exhaustAllowed + maxAwaitTimeMS) and
waited for a stream of replies carrying moreToCome. We answered once with
the flag clear and went back to reading, so every heartbeat failed with
"Server ended moreToCome unexpectedly", destroying the connection and
clearing the pool. MongoDB Compass showed this as a connect/disconnect loop
once per heartbeat.
We do not implement streaming hello, so we must not claim to. Omitting the
field keeps monitoring on the polling path, and agrees with the
maxWireVersion 8 we report: streaming hello arrived in wire version 9.
The existing e2e files all passed against the broken server — they issue
their commands and exit before the second heartbeat — so e2e5 watches SDAM
heartbeats on an idle connection instead.
Also renames mongo-light to mongo-lite throughout (binary, log messages,
docs, gitVersion). Unrelated to the fix above, but squashed in at request
rather than left as a commit whose message described only the fix.
The TTL feature commit documented what the option does but left the
limits sections stale. v1 limits now names the sweep cost (a full walk of
every TTL index entry, under the write lock for the whole pass), so the
interval reads as the tuning knob it is; "Not (yet) implemented" gains
collMod, with the consequence — drop and re-create to change an expiry.
The TTL bullet trades its collMod sentence for the two codes a user
actually hits (IndexOptionsConflict 85 on a changed expiry,
InvalidIndexSpecificationOption 197 on {_id: 1}), the features bullet
mentions the sweeper, and quick start shows a createIndex with
expireAfterSeconds so the feature is visible without reading down.
createIndex({expireAt: 1}, {expireAfterSeconds: 60}) now deletes a
document once its indexed date is that many seconds old.
index.zig carries the option: Index.ttl (?i64), parsed from
expireAfterSeconds (int32/int64/integral double, within MongoDB's
[0, 2147483647]; 0 means "expire at the stored instant"), emitted by
spec_pairs and so persisted through the log and reported by listIndexes,
and compared by spec_equal — a same-name re-create with a different
expiry stays IndexOptionsConflict, as in MongoDB, since collMod does not
exist here. The bound keeps the value an int32 on the wire and makes the
emission cast unconditional. TTL is single-field only (a compound key is
error.TtlOnCompoundIndex); an absent option parses to null, so every
existing log record reparses unchanged.
db.zig sweeps: Engine.ttl_sweep(now_ms) walks each TTL index's entries
and deletes through the ordinary remove path, so an expiry is logged and
fsynced like any other write and holds across a restart. Expired ids are
duped before removal — remove frees the docs-map key that Entry.id
aliases — then sorted and deduped, because one document can be expired
by several entries (an array of dates expires on its earliest member,
which multikey expansion gives for free) or by several TTL indexes.
Selecting entries is a type test rather than a range lookup: bson
compare order ranks datetime above null, numbers and strings, so a
datetime upper bound would also select every value of a lesser type. A
sweep that deleted something checks the compaction threshold, since a
TTL-only workload never reaches the one in upsert.
commands.zig maps the new spec errors: CannotCreateIndex (67) for a
compound key or an out-of-range expiry, and InvalidIndexSpecificationOption
(197) for an expiry on {_id: 1}, which the idempotent _id no-op would
otherwise swallow.
server.zig runs the monitor as a member of the connection group, so the
existing group.cancel tears it down; it sleeps first, then sweeps under
the write lock, and logs rather than dies on a sweep failure.
--ttl-sweep-secs sets the interval (default 60, 0 leaves it unspawned).
Expiry is coarse by design, as in MongoDB: a document stays visible
until the next sweep, and a non-date value at the indexed path never
expires. Unit tests cover the spec round-trip and every rejection, the
sweep (inclusive cutoff, string/missing/future values untouched, second
sweep a no-op) and its survival of a reopen, and two TTL indexes over
one collection. e2e4.js exercises it through the Node driver.
Cleanup pass over the secondary-index feature.
add_doc is now the one entry-commit path. create_index and
build_all_indexes each hand-rolled build -> check_unique -> reserve ->
insert, and had already drifted on whether multikey is set before or
after the unique check; add_doc gained an enforce_unique flag so the
rebuild path keeps its tolerate-and-warn behavior. reserve_for and
insert_entries are now the only way the engine touches Index.entries.
One definition each for: prefix comparison and the prefix binary
searches (prefix_order + std.sort), the cartesian-product odometer
(advance_choice), the spec pair list (write_spec builds on spec_pairs,
so the log format and the listIndexes reply share one schema), the _id
clause parser (plan_id reuses analyze_clause), key-pattern direction
(index.descending, which desc_dir already disagreed with on non-numeric
values), option truthiness (query.truthy), the E11000 message, and
index-removal-by-name (Collection.find_index/remove_index). Key-pattern
matching moved out of the dispatcher into index.find_by_key_pattern.
Dead or redundant: ParallelArraysError, the unread `dropped` counter,
insert_entries' discarded gpa, a third pass computing multikey, the
has_id/is_id_index flag pair, Plan.key_len (always lookup_keys[0].len,
now a method), first_match_consumed (now stages = stages[1..]).
Cheaper hot paths: remove_id compacts in one pass instead of an
orderedRemove per hit; Plan.search skips the sort/dedupe when neither
multikey nor multiple lookup keys can produce a repeat; the _id fast
path reuses one scratch key buffer (bson.write_serialized_value); the
plan loop uses the bound collection instead of re-resolving it through
two hash lookups per candidate.
Behavior is unchanged except that dropping plan_id's fixed 16-clause
buffer enables the _id fast path on filters that previously exceeded it.
README documents supported key patterns, unique/sparse/multikey behavior,
planner rules (multikey two-bound range fallback, sparse/null bail, the
_id fast-path guards), and v1 limits plus the two pre-existing issues the
work surfaces (drop-collection resurrection, compact log_bytes).
e2e3.js exercises createIndex/getIndexes/dropIndex/dropIndexes, the
unique-constraint 11000 path, compound, sparse, and descending indexes
through the official Node driver.
Adds the three driver commands, parameterized E11000 messages (engine
dup_index carries the index name into writeErrors), the scan_matching
planner wiring (_id fast path → index plan → scan) and the first-$match
aggregate pushdown. The equivalence test (mixed-type corpus x 27 filters,
non-sparse and sparse indexes) drove out three real bugs: the two-bound
range under-approximation on multikey indexes (fall back to scan), a
dangling single-value option array in the planner, and update-time
unique violations now reporting writeErrors instead of corrupting state.
Collection gains an indexes list; evict_doc removes entries at the single
document-death chokepoint; upsert does build → check → reserve → log →
evict → publish so entry insertion after the append is infallible and a
rejected unique write never reaches the log. Replay registers empty
indexes from create/drop records (types 3/4) and Engine.open rebuilds them
from live docs. compact re-emits index-create records. Engine gains
create_index/drop_index and dup_index for E11000 naming.
Adds src/index.zig with the full secondary-index machinery: entry
generation mirroring field_matches (array value + elements), BSON-order
sorted entries with binary search, compound prefix and range lookups,
unique/sparse options, the query planner (longest equality/$in run +
optional range, $in cartesian cap, sparse/null bail), and the _id_ fast
path guarded against serialization-ambiguous values (numbers, strings,
symbols, codes, opaque payloads).
query.collect_values is now pub so entry generation can mirror it exactly.
storage.zig gains record_type_index_create/drop; lib.zig exports index.
The Node driver sends {field: /re/} as a BSON regex element (type 0x0B),
which the matcher previously only handled via the $regex operator form;
and dot paths with numeric segments (tags.0) were ignored because array
descent only recursed into embedded docs. Both are part of standard
MongoDB query semantics and were caught by the driver e2e suite.
server: unbounded Io async limit so the accept loop never wedges, and
treat header-read failures (client RST on pool teardown) as clean
disconnects. With the default cpu_count-1 limit, groupAsync's eager
fallback ran connection handlers inline on the accept-loop fiber once
that many connections were alive, stalling accept() and timing out
handshakes for further clients.
Add tests/e2e/: official driver CRUD, concurrency, and kill -9 recovery
suites (29 + 2 + 3 checks), plus unit tests for the query fixes.
The original dispatch restructure placed the unlock defers inside switch
prongs, where Zig runs them when the prong block exits — immediately after
acquisition. All commands therefore ran with no lock at all, which the new
stress test caught deterministically (two writers inside Engine.insert at
once). Lock acquisition now happens in the prongs and the command body runs
via dispatch_impl, so the defers (still prong-scoped) release the lock only
after the command finishes.
