Groundwork for M0: the node arena and overflow slab are about to move into an
mmap'd data file where a write to a page belonging to the last durable
checkpoint has to copy that page first (PLAN amendment A1). Two changes make
that a small commit rather than a sixty-site one, plus the reformat of this
file (see the preceding style commit for why it rides along here).
Accessors. Every read of a node page now goes through page(), every write
through page_mut(), and every overflow read through ovf(); nothing else touches
nodes.items or overflow.items. Which of the 55 sites mutate was decided by the
compiler rather than by inspection -- page() returns *const Node, so every
mutating site failed to compile until flipped -- and the result is that the
copy-on-write hook has exactly one home. Records the rule COW will impose
(never hold a *Node across a page_mut of the same id) and the audit showing
today's callers already comply.
Comptime layout asserts. These structures are about to become an on-disk
format, and nothing pinned them. Pinning also surfaced that @sizeOf(Slot) is
32, not the 20 its 160 declared bits suggest -- the backing integer's 16-byte
alignment rounds it up, so 12 of every 32 slot bytes are padding and a node
holds 127 slots where 203 would fit. Pinned, deliberately not fixed: narrowing
the slot changes the fanout and so the on-disk shape of every index, which
belongs in the commit that reshapes leaf records.
reserve_for. The old bound stood in for "levels a batch can add" with n/8,
which is ~125 levels for a 1000-entry batch and demands ~528 MiB of headroom.
Growing by g levels needs at least 2^g entries, so log2_ceil(n+1)+1 bounds it,
giving ~70 MiB for that batch. Harmless as ArrayList capacity; real file growth
once the arena is file-backed. Overrunning the reservation is a buffer overrun
on a path that has already appended to the log and cannot report failure, so
alloc_node and store_record now assert, using assert.zig so the checks survive
ReleaseFast. Mutation-checked by dropping the reservation entirely: six tests
go red with the new message. Worth noting the assert guards the allocation, not
the arithmetic -- ensureUnusedCapacity over-allocates, so a slightly-too-small
bound is masked until the reservation becomes exact.
build.zig gains a `fuzz` step. spill, spill2, stress and fuzz_split were in no
build step and are not in lib.zig's test block, so `zig build test` could not
see an API break in the only coverage for records past the inline limit and for
randomized split/remove interleavings -- exactly what this work puts at risk.
Prose and benchmark tables use MultiforaDB; the binary, the CLI usage
line, the log-message prefix and the default database file use
multiforadb.
Two consequences worth noting:
- build.zig.zon's fingerprint is derived from the package name, so it
had to change with it (Zig refuses to build otherwise). A consumer
pinning this package by fingerprint needs updating.
- the default --db path is now multiforadb.log, and getCmdLineOpts
reports it as dbpath. An existing mongo-lite.log has to be passed
explicitly with --db.
The e2e harness abbreviated the old name as ML_; that is now MFDB_,
including the documented ML_BIN override (MFDB_BIN) and the scratch
file names. MD_ (mongod) is untouched.
compare-run.sh spawned the server by absolute path under a
sandbox/mongo-lite directory that no longer exists; that block already
runs from tests/e2e, so it uses a relative path now.
The archived reports under tests/e2e/results/ keep the old name: they
record what the old binary measured.
Two streams of work land together: they are interleaved in storage.zig
and db.zig and only build as a unit.
Already in the working tree before this session:
- ReleaseFast as the default zig build (Debug was 10-200x slower)
- group commit: one fsync per write command instead of per document
- plan_id returned a pointer to a stack temporary; ReleaseFast read
garbage and silently broke findOne({_id: ObjectId})
- perf suite: big.js, compare.js, compare-run.sh, e2e6.js
Phase 1 performance work:
Record integrity hash CRC32 -> XxHash3. std.hash.Crc32 is the
table-driven byte-at-a-time Crc32IsoHdlc, measured at 408 MB/s against
XxHash3's 31 GB/s: 38us versus 0.5us on a 16 KiB document, which was
about two thirds of the entire bulk-insert cost. The record header
grows from u32 crc to u64 hash (header_len 20 -> 24), a breaking
format change. Bulk insert 260 -> 700 MB/s, reopen 1.1 -> 0.5s.
Compaction fsynced once per live document, because Log.open leaves
defer_sync false and compact never set it. It now issues one sync for
the whole rewrite, before the rename that publishes it.
Compaction triggers on the share of the log that is garbage
(live_docs/dead_docs, maintained at evict_doc, the single point where
a document dies) rather than on bytes appended. A fixed byte count is
wrong in both directions: a 1 GB bulk load holds no garbage at all yet
would compact ~64 times under the 16 MiB default, rewriting 1 GB each
time, while a small collection rewritten in place accumulates garbage
indefinitely without ever reaching the count. Pure inserts now never
compact, and the file stays near 1.25x the live data. Bulk load at the
default threshold: 41.6 -> 702.7 MB/s.
remove() never called maybe_compact, so a delete-heavy workload grew
the log without bound.
e2e6's compaction check required the file to bloat past 30 MiB before
being reclaimed, which encoded the old policy and failed on strictly
better behaviour (ends at 15.8 MB against ~12 MB live, was ~30 MB). It
now asserts the file ends near the live size and peaked well above it,
which does not depend on when the trigger fires. Sampling interval 50
-> 10ms: the operations now finish inside the old window.
Verified: 70 unit tests under both ReleaseFast and ReleaseSafe, e2e
29/16/17/3/2 checks, the crash-a/kill -9/crash-b pair, and e2e6 72/72
across three consecutive runs.
Advertising topologyVersion in the hello reply is what tells a driver the
server speaks the streaming (awaitable) hello protocol — in the Node driver
it is the only condition checked. From the second heartbeat on, the driver
then monitored with an exhaust hello (exhaustAllowed + maxAwaitTimeMS) and
waited for a stream of replies carrying moreToCome. We answered once with
the flag clear and went back to reading, so every heartbeat failed with
"Server ended moreToCome unexpectedly", destroying the connection and
clearing the pool. MongoDB Compass showed this as a connect/disconnect loop
once per heartbeat.
We do not implement streaming hello, so we must not claim to. Omitting the
field keeps monitoring on the polling path, and agrees with the
maxWireVersion 8 we report: streaming hello arrived in wire version 9.
The existing e2e files all passed against the broken server — they issue
their commands and exit before the second heartbeat — so e2e5 watches SDAM
heartbeats on an idle connection instead.
Also renames mongo-light to mongo-lite throughout (binary, log messages,
docs, gitVersion). Unrelated to the fix above, but squashed in at request
rather than left as a commit whose message described only the fix.
