Every reply came back in a single batch with `cursor.id = 0`, `getMore` was a
stub answering an empty `nextBatch` on the literal namespace `test.$cmd`, and
nothing read `batchSize`. That caps the useful collection size at what fits in
one 48 MiB message, which is the opposite of the tens-of-GB target and the
reason M0 made whole-index scans stream: the streaming candidate generator
existed with no consumer that could suspend.
## What a cursor is allowed to remember
A cursor holds no lock between requests, so everything it saves has to survive
arbitrary concurrent mutation. Nothing here is a pointer, and the two things
that look like stable addresses are not: `reset_tree` re-creates node ids 0 and
1 as different nodes, and `rebuild_collection` moves every document. Three
sources, chosen by query shape, each with a different memory contract:
- **stream** -- an index-ordered walk resumed from a `(key, off)` anchor plus a
`(leaf, slot)` hint. O(key) state, so this is what lets a cursor walk a
collection larger than memory. Survives a rebuild, because a repack changes
no key.
- **offsets** -- the matched slab offsets a narrowed plan already materialized,
8 bytes each. Killed by a rebuild with `QueryPlanKilled`, because those
offsets now name unrelated bytes.
- **buffered** -- canonical BSON copies, for a sort no index provides and for
aggregate/listing output. Depends on nothing, which is what lets a listing
hold a cursor over a `$cmd.*` namespace no collection backs.
`Collection.layout_epoch` and `Index.epoch` are the invalidation tokens, both
checked as error returns rather than assertions since a client reaches them by
keeping a cursor open across maintenance.
## Resume
`resume_forward`/`resume_reverse` are O(1) while the hint holds and fall back to
an exact-order band walk bounded by `resume_walk_max`. Without the hint, `seek`
lands at the *start* of an equal-key band, so `sort({status: 1})` over three
distinct values across 10M documents would cost ~5e10 comparisons to drain.
Two hazards found by draining a collection while writing to it, neither
predictable from reading the code:
- A deleted anchor must resume at its *band position*, or the rest of an
equal-key band is silently dropped -- most of the collection on a
low-cardinality index. Hence `band_index`.
- On a **unique** index a same-key entry can only be the anchor rewritten, so
resuming at it returned updated documents twice. Observed as duplicate `_id`s
while updating underneath a drain.
## Protocol
Measured against mongod 8.3.7 rather than recalled, which corrected three
assumptions: a bare `getMore` does *not* inherit the find's `batchSize` (4998 of
5000 documents come back), a namespace mismatch is `Unauthorized` (13) not
`CursorNotFound`, and `CursorInUse` is 143 not 12051.
`internalQueryFindCommandBatchSize` is 101, `cursorTimeoutMillis` 600000,
`clientCursorMonitorFrequencySecs` 4.
The rule everything follows is **never look ahead**: a batch that met its target
leaves the cursor open even when the source is in fact exhausted, so four
documents at `batchSize: 2` take three commands. `limit` acts as an EOF source,
which is what makes `batchSize == limit` close in one round trip. `skip` is
consumed once. `batchSize: 0` returns an empty batch with a live cursor.
Cursor ids are `(nonce << 20) | slot`, always positive. The nonce is not
decoration: without it a recycled slot serves one client another's documents.
Cursors are not connection-pinned, since the driver spec allows a `getMore` on
any connection to the same server; they end at exhaustion, `killCursors`, or the
idle sweep (a second monitor fiber, separate from the TTL one because the
cadences differ by an order of magnitude and a TTL failure must not stop
reclamation). The registry is fixed-capacity and evicts the least recently used
cursor, whose client sees the same 43 an idle timeout gives.
Fixed alongside, because cursors are what expose them:
- `listCollections` reported `"<db>."` with an *empty* collection part, which
makes the driver throw client-side -- so it would have broken the moment its
cursor stopped being id 0. Now `<db>.$cmd.listCollections`, as mongod uses.
- `count` ignored `skip` and `limit` entirely.
- `wire.end_message` now bounds a reply by the 48 MiB we advertise rather than
by `maxInt(u32)`; a reply past what we told the client to expect is not a large
reply, it is a desynchronized connection.
- Two `codeName` strings were wrong: 72 is `InvalidOptions` (MongoDB has no
`InvalidArgument`), and 40324 reports as `Location40324`.
## Verification
Unit 160/160 in ReleaseFast and ReleaseSafe; `tests/e2e/e2e7.js` adds 86 cursor
checks across five phases (batching/lifecycle/errors, streaming across churn,
aggregate+listings+count, expiry+capacity, restart) and is self-contained
because cursor behaviour is only observable with non-default flags. No
regressions: e2e 49, e2e3 16, e2e4 17, e2e2 2, e2e6 72. Spec 168 pass / 124
fail, +5 against the previous scorecard.
Mutation-checked, per the repo's second ground rule: `hint_slot + 1`, the
`band_index` off-by-one, both epoch bumps, the id nonce, the at-least-one-
document rule, and `stream_shape` returning null each turn the intended test
red. One claim was withdrawn rather than kept -- swapping `std.mem.order` for
`cmp_prefix` in the band walk changes nothing observable, so the comment now
says so instead of asserting a check that does not hold.
Prose and benchmark tables use MultiforaDB; the binary, the CLI usage
line, the log-message prefix and the default database file use
multiforadb.
Two consequences worth noting:
- build.zig.zon's fingerprint is derived from the package name, so it
had to change with it (Zig refuses to build otherwise). A consumer
pinning this package by fingerprint needs updating.
- the default --db path is now multiforadb.log, and getCmdLineOpts
reports it as dbpath. An existing mongo-lite.log has to be passed
explicitly with --db.
The e2e harness abbreviated the old name as ML_; that is now MFDB_,
including the documented ML_BIN override (MFDB_BIN) and the scratch
file names. MD_ (mongod) is untouched.
compare-run.sh spawned the server by absolute path under a
sandbox/mongo-lite directory that no longer exists; that block already
runs from tests/e2e, so it uses a relative path now.
The archived reports under tests/e2e/results/ keep the old name: they
record what the old binary measured.
compare-run.sh answers "how do we compare to MongoDB"; it says nothing
about whether a change made things better or worse than last week. Add a
harness that records each run and diffs it against the previous one.
bench-run.sh wraps compare-run.sh, adds a concurrent durable-write
comparison (concurrent.js: N clients each doing sequential insertOne with
{w:1, j:true}, exercising the group-commit path under real contention),
writes a versioned name<TAB>value report to results/bench-<timestamp>.txt,
and prints a diff of our numbers against results/bench-latest.txt.
Also:
- compare-run.sh polled with fixed sleeps, which are flaky once earlier
benchmark phases have warmed the machine; both servers now wait on a
real driver connection instead.
- a dispatch error only reached the client as a generic InternalError,
with nothing on the server side naming the failing command; log the
connection, command and error name before replacing the reply.
Two streams of work land together: they are interleaved in storage.zig
and db.zig and only build as a unit.
Already in the working tree before this session:
- ReleaseFast as the default zig build (Debug was 10-200x slower)
- group commit: one fsync per write command instead of per document
- plan_id returned a pointer to a stack temporary; ReleaseFast read
garbage and silently broke findOne({_id: ObjectId})
- perf suite: big.js, compare.js, compare-run.sh, e2e6.js
Phase 1 performance work:
Record integrity hash CRC32 -> XxHash3. std.hash.Crc32 is the
table-driven byte-at-a-time Crc32IsoHdlc, measured at 408 MB/s against
XxHash3's 31 GB/s: 38us versus 0.5us on a 16 KiB document, which was
about two thirds of the entire bulk-insert cost. The record header
grows from u32 crc to u64 hash (header_len 20 -> 24), a breaking
format change. Bulk insert 260 -> 700 MB/s, reopen 1.1 -> 0.5s.
Compaction fsynced once per live document, because Log.open leaves
defer_sync false and compact never set it. It now issues one sync for
the whole rewrite, before the rename that publishes it.
Compaction triggers on the share of the log that is garbage
(live_docs/dead_docs, maintained at evict_doc, the single point where
a document dies) rather than on bytes appended. A fixed byte count is
wrong in both directions: a 1 GB bulk load holds no garbage at all yet
would compact ~64 times under the 16 MiB default, rewriting 1 GB each
time, while a small collection rewritten in place accumulates garbage
indefinitely without ever reaching the count. Pure inserts now never
compact, and the file stays near 1.25x the live data. Bulk load at the
default threshold: 41.6 -> 702.7 MB/s.
remove() never called maybe_compact, so a delete-heavy workload grew
the log without bound.
e2e6's compaction check required the file to bloat past 30 MiB before
being reclaimed, which encoded the old policy and failed on strictly
better behaviour (ends at 15.8 MB against ~12 MB live, was ~30 MB). It
now asserts the file ends near the live size and peaked well above it,
which does not depend on when the trigger fires. Sampling interval 50
-> 10ms: the operations now finish inside the old window.
Verified: 70 unit tests under both ReleaseFast and ReleaseSafe, e2e
29/16/17/3/2 checks, the crash-a/kill -9/crash-b pair, and e2e6 72/72
across three consecutive runs.
Advertising topologyVersion in the hello reply is what tells a driver the
server speaks the streaming (awaitable) hello protocol — in the Node driver
it is the only condition checked. From the second heartbeat on, the driver
then monitored with an exhaust hello (exhaustAllowed + maxAwaitTimeMS) and
waited for a stream of replies carrying moreToCome. We answered once with
the flag clear and went back to reading, so every heartbeat failed with
"Server ended moreToCome unexpectedly", destroying the connection and
clearing the pool. MongoDB Compass showed this as a connect/disconnect loop
once per heartbeat.
We do not implement streaming hello, so we must not claim to. Omitting the
field keeps monitoring on the polling path, and agrees with the
maxWireVersion 8 we report: streaming hello arrived in wire version 9.
The existing e2e files all passed against the broken server — they issue
their commands and exit before the second heartbeat — so e2e5 watches SDAM
heartbeats on an idle connection instead.
Also renames mongo-light to mongo-lite throughout (binary, log messages,
docs, gitVersion). Unrelated to the fix above, but squashed in at request
rather than left as a commit whose message described only the fix.
createIndex({expireAt: 1}, {expireAfterSeconds: 60}) now deletes a
document once its indexed date is that many seconds old.
index.zig carries the option: Index.ttl (?i64), parsed from
expireAfterSeconds (int32/int64/integral double, within MongoDB's
[0, 2147483647]; 0 means "expire at the stored instant"), emitted by
spec_pairs and so persisted through the log and reported by listIndexes,
and compared by spec_equal — a same-name re-create with a different
expiry stays IndexOptionsConflict, as in MongoDB, since collMod does not
exist here. The bound keeps the value an int32 on the wire and makes the
emission cast unconditional. TTL is single-field only (a compound key is
error.TtlOnCompoundIndex); an absent option parses to null, so every
existing log record reparses unchanged.
db.zig sweeps: Engine.ttl_sweep(now_ms) walks each TTL index's entries
and deletes through the ordinary remove path, so an expiry is logged and
fsynced like any other write and holds across a restart. Expired ids are
duped before removal — remove frees the docs-map key that Entry.id
aliases — then sorted and deduped, because one document can be expired
by several entries (an array of dates expires on its earliest member,
which multikey expansion gives for free) or by several TTL indexes.
Selecting entries is a type test rather than a range lookup: bson
compare order ranks datetime above null, numbers and strings, so a
datetime upper bound would also select every value of a lesser type. A
sweep that deleted something checks the compaction threshold, since a
TTL-only workload never reaches the one in upsert.
commands.zig maps the new spec errors: CannotCreateIndex (67) for a
compound key or an out-of-range expiry, and InvalidIndexSpecificationOption
(197) for an expiry on {_id: 1}, which the idempotent _id no-op would
otherwise swallow.
server.zig runs the monitor as a member of the connection group, so the
existing group.cancel tears it down; it sleeps first, then sweeps under
the write lock, and logs rather than dies on a sweep failure.
--ttl-sweep-secs sets the interval (default 60, 0 leaves it unspawned).
Expiry is coarse by design, as in MongoDB: a document stays visible
until the next sweep, and a non-date value at the indexed path never
expires. Unit tests cover the spec round-trip and every rejection, the
sweep (inclusive cutoff, string/missing/future values untouched, second
sweep a no-op) and its survival of a reopen, and two TTL indexes over
one collection. e2e4.js exercises it through the Node driver.
The Node driver sends {field: /re/} as a BSON regex element (type 0x0B),
which the matcher previously only handled via the $regex operator form;
and dot paths with numeric segments (tags.0) were ignored because array
descent only recursed into embedded docs. Both are part of standard
MongoDB query semantics and were caught by the driver e2e suite.
server: unbounded Io async limit so the accept loop never wedges, and
treat header-read failures (client RST on pool teardown) as clean
disconnects. With the default cpu_count-1 limit, groupAsync's eager
fallback ran connection handlers inline on the accept-loop fiber once
that many connections were alive, stalling accept() and timing out
handshakes for further clients.
Add tests/e2e/: official driver CRUD, concurrency, and kill -9 recovery
suites (29 + 2 + 3 checks), plus unit tests for the query fixes.
