New src/pager.zig, engine-unused at this commit: the structures move onto it in
the commits that follow, and landing it alone keeps that change reviewable.
The file is an array of 4 KiB pages with a tail-bump extent allocator. PLAN
D6.1 asked for a region table; it cannot be one, because there is a node arena
per index and a document slab per collection, so the region count is dynamic and
unbounded and N contiguous regions cannot all grow at the tail. One page array
means exactly one growth path, so the write-then-extend discipline lives in
exactly one place, and `ls`/`du` stay honest for the backup story.
The mapping is a PROT_NONE anonymous NORESERVE reservation that new file-backed
suffixes are mmap'd into with MAP_FIXED. That is one VMA and zero committed
pages, and it means **the base never moves for the life of the process**, so a
pointer handed out before a growth is still valid after it. The
ArrayList-backed arena this replaces could not promise that -- the promoted-key
scratch buffer in index.zig exists solely to work around it, and phase8 records
a dangling-slab-pointer bug of exactly that shape.
Growth is `setLength` and *then* `mmap`, never the reverse: a store into a
mapped page past end-of-file raises SIGBUS, which no error path can catch. The
accessors assert against `mapped_pages`, so a violation is a panic with a
message instead of a signal.
`page_mut` is deliberately the only way to obtain a writable page. Copy-on-write
hooks in there (commit 12), and funnelling every write through one function is
what makes that a change of one body rather than of every caller.
`sync` is msync + fsync and only the checkpoint calls it. Between checkpoints
dirty pages may sit in the page cache indefinitely, because recovery is
`image + replay(seq > watermark)` and the image's pages are never written
*differently* -- which is what keeps the write path at exactly one fsync, the
WAL's (PLAN amendment A1).
This is the one place in src/ that reaches for std.posix, against the house
style, and the module comment says why: std.Io.File.MemoryMap prefaults by
default, exposes no NORESERVE/FIXED/address hint so it cannot express a
reservation, and its setLength is mremap on Linux and unsupported on darwin.
Mutation-checked, all four red: remapping the prefix at a kernel-chosen address;
dropping the uuid comparison (which is what stops one database's log being
replayed onto another's checkpoint); dropping the header hash comparison; and
moving setLength after mmap. An empty file is treated as absent rather than as
corruption, so a create that died before its header landed still opens.
Prose and benchmark tables use MultiforaDB; the binary, the CLI usage
line, the log-message prefix and the default database file use
multiforadb.
Two consequences worth noting:
- build.zig.zon's fingerprint is derived from the package name, so it
had to change with it (Zig refuses to build otherwise). A consumer
pinning this package by fingerprint needs updating.
- the default --db path is now multiforadb.log, and getCmdLineOpts
reports it as dbpath. An existing mongo-lite.log has to be passed
explicitly with --db.
The e2e harness abbreviated the old name as ML_; that is now MFDB_,
including the documented ML_BIN override (MFDB_BIN) and the scratch
file names. MD_ (mongod) is untouched.
compare-run.sh spawned the server by absolute path under a
sandbox/mongo-lite directory that no longer exists; that block already
runs from tests/e2e, so it uses a relative path now.
The archived reports under tests/e2e/results/ keep the old name: they
record what the old binary measured.
Follow-up hardening on the group-commit work from ecd28d9 and c8d547f. The
signal -> broadcast fix and the append-drained wakeup under commit_lock were
correct but incomplete; each of the three defects below could lose or corrupt
data that had already been acknowledged.
- log_append's cleanup defer took commit_lock with `lock(...) catch {}` and
then unlocked unconditionally. Mutex.lock is Cancelable!void and
Mutex.unlock treats an already-unlocked mutex as `unreachable`, so a
cancellation there (client disconnect, shutdown) released a mutex the fiber
never held: a panic in ReleaseSafe and silent memory corruption in the
default ReleaseFast build. A cleanup path must not be a cancellation point,
so it uses lockUncancelable.
- Engine.commit waited with `catch return`, which returns *success* from
!void. A failed wait therefore told the caller its write was on disk and the
dispatch epilogue replied ok without a seal or an fsync -- the same failure
class c8d547f fixed, reached through the error path instead of the happy
one. The follower wait now propagates; the leader's drain is uncancelable,
since once `committing` is set every other writer is parked behind it and
the drain is bounded anyway.
- Two compactions could run at once. They share one `<log>.tmp` path and each
ends in a rename onto the log, so one truncates and rewrites the file the
other is about to publish, and then that one renames whatever it finds over
the live log. compact now claims an atomic `compacting` slot and a second
caller returns; the guard sits on the resource rather than in take_compact,
so direct callers (tests included) are covered too. compact_pending became
atomic while we were there: note_compact sets it under a *collection* lock
and the epilogue read it under none.
Also in compaction: the tmp file is opened with a new Log.create that
truncates. Log.open keeps an existing file's bytes and only rewinds end_pos to
the header, so a longer tmp left by a crashed or retried rewrite kept its
tail -- and those trailing blocks are intact and hash-correct, so replay
applied them as live records once the rename published the file, resurrecting
deleted documents. The `deleteFile ... catch {}` that used to stand in for
this is gone, and with it a swallowed error the invariant rested on. The
retry loop is bounded at 8 attempts (each one rewrites the whole log before
the seq check can reject it, so an unbounded retry livelocks under sustained
writes); giving up re-arms the request instead of failing the write.
A compaction failure no longer fails the write whose epilogue triggered it:
the write is durable by then, so the error is reported and the request
re-armed rather than turned into an error the client retries.
Assertions: db.zig, commands.zig and storage.zig had none, which is why every
defect in this series was found by a stress run rather than at the moment of
corruption. src/assert.zig adds an assert that survives ReleaseFast --
std.debug.assert lowers to `unreachable`, which in this project's default
build is not a skipped check but a promise to the optimizer, exactly the wrong
lowering for a durability invariant that might be false. Eleven of them now
cover the commit watermark, the in-flight append count, the compaction
snapshot, and the live/dead counters (whose u64 subtraction would otherwise
underflow into a live count that suppresses compaction forever).
cmd_find asserts that a missing collection implies no matches instead of
silently emitting an empty page for a query that did match.
Dead state removed: Log.defer_sync was never read (only written once by
compact), yet its doc comment instructed callers to follow a defer_sync
protocol that no longer exists and has no effect if followed. Engine's
begin_batch/end_batch were both `_ = self`, so three call sites announced a
batch boundary that wasn't there. Both are gone and the comments now describe
the real contract: appends never sync, Log.sync is the only commit point.
Tests: Log.create's truncation is pinned by a storage test that replays after
reusing a path, and the compaction guard by a db test that drives the flag
directly -- both confirmed to fail without their fix. The threaded test added
alongside them is a smoke test only, and says so: both races have windows too
narrow to hit reliably (compact_snapshot_coll holds each collection's write
lock while snapshotting, so two compactions serialize there and an insert
cannot re-arm compact_pending meanwhile), and it passes with the guard
removed.
Includes an unrelated fix that was already in the working tree: slab_append
held a pointer into slab.items across an append to that same list, which
could dangle after a realloc and corrupt the new segment's start offset.
Verified: unit suite in ReleaseFast/ReleaseSafe/Debug; e2e, e2e2 concurrent,
e2e3, e2e4, e2e5, e2e6 (72/72) and the kill -9 crash pair, with no assertion
firing anywhere.
Advertising topologyVersion in the hello reply is what tells a driver the
server speaks the streaming (awaitable) hello protocol — in the Node driver
it is the only condition checked. From the second heartbeat on, the driver
then monitored with an exhaust hello (exhaustAllowed + maxAwaitTimeMS) and
waited for a stream of replies carrying moreToCome. We answered once with
the flag clear and went back to reading, so every heartbeat failed with
"Server ended moreToCome unexpectedly", destroying the connection and
clearing the pool. MongoDB Compass showed this as a connect/disconnect loop
once per heartbeat.
We do not implement streaming hello, so we must not claim to. Omitting the
field keeps monitoring on the polling path, and agrees with the
maxWireVersion 8 we report: streaming hello arrived in wire version 9.
The existing e2e files all passed against the broken server — they issue
their commands and exit before the second heartbeat — so e2e5 watches SDAM
heartbeats on an idle connection instead.
Also renames mongo-light to mongo-lite throughout (binary, log messages,
docs, gitVersion). Unrelated to the fix above, but squashed in at request
rather than left as a commit whose message described only the fix.
Adds src/index.zig with the full secondary-index machinery: entry
generation mirroring field_matches (array value + elements), BSON-order
sorted entries with binary search, compound prefix and range lookups,
unique/sparse options, the query planner (longest equality/$in run +
optional range, $in cartesian cap, sparse/null bail), and the _id_ fast
path guarded against serialization-ambiguous values (numbers, strings,
symbols, codes, opaque payloads).
query.collect_values is now pub so entry generation can mirror it exactly.
storage.zig gains record_type_index_create/drop; lib.zig exports index.
