Encodes a Value so that std.mem.order over the bytes reproduces
bson.compare exactly. This is the foundation for the encoded-key index:
it lets an index binary-search, range-scan and eventually be stored as
raw bytes, instead of carrying Value trees whose every comparison chases
pointers into a different document's arena.
Layout is [rank + 1] then a self-delimiting payload; the +1 keeps 0x00
out of the tag space so it can terminate variable-length payloads.
The parts that are easy to get wrong, and why they are the way they are:
Numbers encode the f128 that compare already widens int32, int64 and
double to -- exactly, for all three. So int32 1, int64 1 and double 1.0
produce identical bytes, which is the cross-type equality that numeric
index lookups need, and is precisely what value_fast_path_safe exists
today to work around. Negatives are bit-inverted and positives get the
sign bit set, making the IEEE order lexicographic. -0.0 normalizes to
+0.0 (they compare equal) and every NaN encodes as all-ones (compare
makes NaN greatest and all NaNs equal).
Byte strings escape 0x00 as 00 FF and terminate with 00 00. A BSON
string may contain NUL, so a bare terminator would be ambiguous;
escaping fixes ordering at the same time, since a real NUL then sorts
above the terminator and any byte >= 01 does too. "Shorter is less"
falls out to match std.mem.order, which also gives documents and arrays
their length tie-break for free.
Binary length-prefixes because compare_binary orders by length first,
but opaque_val escapes instead: compare ignores its kind and orders the
data lexicographically, not by length.
Correctness rests entirely on the order equivalence, so it is checked
exhaustively rather than by example: every ordered pair of a corpus
spanning all fifteen ranks and their boundaries (numeric cross-type and
sign, NaN, both zeros, infinities, embedded NULs, prefix relationships,
empty and nested documents and arrays, binary subtypes) is compared both
ways. A second test concatenates two-column keys and checks they
reproduce component-wise order, which is what makes compound keys and
prefix search sound. Verified both fail when escaping is dropped, when
-0.0 is not normalized, when the binary length prefix is wrong, and when
NaN stops being greatest.
Nothing uses the encoding yet; the index still holds Value keys.
Matching allocated an ArrayList per filter field per candidate document,
on the process-wide allocator, to hold what is almost always a single
value. Candidates now collect into a stack buffer that spills to the heap
only for arrays: measured 15.7 -> 12.0ms on a 65,536-document range scan.
The OOM-propagation test moves with it. Its point is that a failed
collection must surface as an error rather than an empty candidate list,
which would make $ne and $exists:false report a match -- a wrong answer
rather than a failed one. That invariant still holds on the spill path, so
the test now uses an array long enough to reach the allocator, and a new
test pins the flip side: the common single-value match now completes
correctly even when the allocator always fails, because it never calls it.
Query operators were dispatched by a chain of up to fourteen mem.eql per
value per document, with $gt/$gte/$lt/$lte re-comparing the operator name
inside the loop over candidate values. Names resolve to an enum once per
filter field. Command dispatch likewise walked a 30-entry table comparing
strings; it is a comptime StaticStringMap now.
Each request built a fresh reply arena and handed its pages straight back.
One reply per connection, reset between requests, keeps them.
countDocuments() arrives as [{$match: F}?, {$group: {_id: <literal>,
n: {$sum: 1}}}], which the general path answered by materializing every
matching document and discarding them all. It is now recognized and
answered from a counting scan: countDocuments({}) 2.3 -> 1.5ms.
The detector is deliberately conservative -- grouping by "$field", summing
a field, an unmodelled accumulator or any extra stage all fall through to
the general path, since those need the documents themselves. A unit test
pins each accept and reject, and the whole count path was checked against
the general one through the real driver, including the shapes that must
not take it.
The filtered range-scan row does not move: it is bound by walking 65,536
documents that each live in their own arena, not by the matcher. That is
Phase 4 work.
Verified: 77 unit tests under ReleaseFast and ReleaseSafe, e2e
29/16/17/3/2, the crash pair, e2e6 72/72.
sort+limit ordered the entire result set to return one page: 65,536
documents sorted to hand back 20. Two independent costs.
The decorate pass built, per document per sort key, an ArrayList of every
value at the path -- but the comparator only ever reads element 0. Added
first_value_at, which mirrors collect_values' traversal exactly (same
order, same depth cutoff) and stops at the first hit, and moved the
decorated values into one flat allocation. That equivalence is the whole
correctness argument, so it is pinned by a test covering dotted paths,
arrays of documents, numeric element addressing, repeated keys, missing
paths and the depth cutoff.
sort_docs_top_k keeps a k-element max-heap instead of ordering
everything: one comparison against the heap root per document, and only
the survivors are ever sorted. cmd_find uses it when the page is at most
a quarter of the matches, where the heap's bookkeeping still pays for
itself, and falls back to a full sort otherwise. It leaves docs[k..]
unordered, which is safe because the page is a prefix of the first k.
cmd_aggregate's $sort is deliberately untouched: a later stage can read
the whole stream, and top-k would silently corrupt the tail.
find({}).sort({_id:-1}).limit(20) over 65,536 x 16 KB documents:
baseline 40.0ms
decorate only (top-k disabled) 23.9ms
decorate + top-k 4.3ms
Correctness checked end to end as well: the limited page is identical to
the prefix of the equivalent full sort. The top-k test compares against a
full sort across ascending, descending and compound keys, for k of 1, 2,
20, n-1, n and n+1, over data with heavy ties; verified it fails when the
heap's child comparison is inverted.
createIndex built the entry array one document at a time, and each
insert kept the array sorted by memmoving the tail -- O(n^2) bytes moved
over a full build, which was the entire cost of the operation. Entries
are now appended unsorted and ordered once (append_doc_entries +
finish_bulk), with uniqueness checked by a single adjacent-pair scan
instead of a binary search per document. build_all_indexes, which runs
for every index on every open, takes the same path.
createIndex over 65,536 documents, measured A/B:
{k: 1} 649ms -> 56ms
{s: 1} unique 678ms -> 53ms
{p: 1, k: -1} 653ms -> 54ms
lookup_range binary-searched only the equality prefix and then scanned
that whole band applying a filter, so a range on the first component of
an index touched every entry in it. Both ends are now binary searches
over the component the array is already sorted on, clamped into the
equality band. Note this does not move the range-scan row in compare.js:
that query filters on p, which has no index there, so it is a collection
scan and belongs to the matcher.
cmd_find passed a hardcoded 0 as the scan limit, so find().limit(n)
materialized the entire collection before slicing. It now stops once the
page is filled, when there is no sort to order the matches first; the
bound covers the skipped prefix because the scan counts matches rather
than returned documents.
lookup_range's bounds are checked by a new randomized test that compares
the result count against a brute-force filter over 600 generated
queries, with values chosen from a small domain so equal keys and the
inclusive/exclusive edges come up constantly. Verified it fails when
either bound is swapped.
Two streams of work land together: they are interleaved in storage.zig
and db.zig and only build as a unit.
Already in the working tree before this session:
- ReleaseFast as the default zig build (Debug was 10-200x slower)
- group commit: one fsync per write command instead of per document
- plan_id returned a pointer to a stack temporary; ReleaseFast read
garbage and silently broke findOne({_id: ObjectId})
- perf suite: big.js, compare.js, compare-run.sh, e2e6.js
Phase 1 performance work:
Record integrity hash CRC32 -> XxHash3. std.hash.Crc32 is the
table-driven byte-at-a-time Crc32IsoHdlc, measured at 408 MB/s against
XxHash3's 31 GB/s: 38us versus 0.5us on a 16 KiB document, which was
about two thirds of the entire bulk-insert cost. The record header
grows from u32 crc to u64 hash (header_len 20 -> 24), a breaking
format change. Bulk insert 260 -> 700 MB/s, reopen 1.1 -> 0.5s.
Compaction fsynced once per live document, because Log.open leaves
defer_sync false and compact never set it. It now issues one sync for
the whole rewrite, before the rename that publishes it.
Compaction triggers on the share of the log that is garbage
(live_docs/dead_docs, maintained at evict_doc, the single point where
a document dies) rather than on bytes appended. A fixed byte count is
wrong in both directions: a 1 GB bulk load holds no garbage at all yet
would compact ~64 times under the 16 MiB default, rewriting 1 GB each
time, while a small collection rewritten in place accumulates garbage
indefinitely without ever reaching the count. Pure inserts now never
compact, and the file stays near 1.25x the live data. Bulk load at the
default threshold: 41.6 -> 702.7 MB/s.
remove() never called maybe_compact, so a delete-heavy workload grew
the log without bound.
e2e6's compaction check required the file to bloat past 30 MiB before
being reclaimed, which encoded the old policy and failed on strictly
better behaviour (ends at 15.8 MB against ~12 MB live, was ~30 MB). It
now asserts the file ends near the live size and peaked well above it,
which does not depend on when the trigger fires. Sampling interval 50
-> 10ms: the operations now finish inside the old window.
Verified: 70 unit tests under both ReleaseFast and ReleaseSafe, e2e
29/16/17/3/2 checks, the crash-a/kill -9/crash-b pair, and e2e6 72/72
across three consecutive runs.
Advertising topologyVersion in the hello reply is what tells a driver the
server speaks the streaming (awaitable) hello protocol — in the Node driver
it is the only condition checked. From the second heartbeat on, the driver
then monitored with an exhaust hello (exhaustAllowed + maxAwaitTimeMS) and
waited for a stream of replies carrying moreToCome. We answered once with
the flag clear and went back to reading, so every heartbeat failed with
"Server ended moreToCome unexpectedly", destroying the connection and
clearing the pool. MongoDB Compass showed this as a connect/disconnect loop
once per heartbeat.
We do not implement streaming hello, so we must not claim to. Omitting the
field keeps monitoring on the polling path, and agrees with the
maxWireVersion 8 we report: streaming hello arrived in wire version 9.
The existing e2e files all passed against the broken server — they issue
their commands and exit before the second heartbeat — so e2e5 watches SDAM
heartbeats on an idle connection instead.
Also renames mongo-light to mongo-lite throughout (binary, log messages,
docs, gitVersion). Unrelated to the fix above, but squashed in at request
rather than left as a commit whose message described only the fix.
createIndex({expireAt: 1}, {expireAfterSeconds: 60}) now deletes a
document once its indexed date is that many seconds old.
index.zig carries the option: Index.ttl (?i64), parsed from
expireAfterSeconds (int32/int64/integral double, within MongoDB's
[0, 2147483647]; 0 means "expire at the stored instant"), emitted by
spec_pairs and so persisted through the log and reported by listIndexes,
and compared by spec_equal — a same-name re-create with a different
expiry stays IndexOptionsConflict, as in MongoDB, since collMod does not
exist here. The bound keeps the value an int32 on the wire and makes the
emission cast unconditional. TTL is single-field only (a compound key is
error.TtlOnCompoundIndex); an absent option parses to null, so every
existing log record reparses unchanged.
db.zig sweeps: Engine.ttl_sweep(now_ms) walks each TTL index's entries
and deletes through the ordinary remove path, so an expiry is logged and
fsynced like any other write and holds across a restart. Expired ids are
duped before removal — remove frees the docs-map key that Entry.id
aliases — then sorted and deduped, because one document can be expired
by several entries (an array of dates expires on its earliest member,
which multikey expansion gives for free) or by several TTL indexes.
Selecting entries is a type test rather than a range lookup: bson
compare order ranks datetime above null, numbers and strings, so a
datetime upper bound would also select every value of a lesser type. A
sweep that deleted something checks the compaction threshold, since a
TTL-only workload never reaches the one in upsert.
commands.zig maps the new spec errors: CannotCreateIndex (67) for a
compound key or an out-of-range expiry, and InvalidIndexSpecificationOption
(197) for an expiry on {_id: 1}, which the idempotent _id no-op would
otherwise swallow.
server.zig runs the monitor as a member of the connection group, so the
existing group.cancel tears it down; it sleeps first, then sweeps under
the write lock, and logs rather than dies on a sweep failure.
--ttl-sweep-secs sets the interval (default 60, 0 leaves it unspawned).
Expiry is coarse by design, as in MongoDB: a document stays visible
until the next sweep, and a non-date value at the indexed path never
expires. Unit tests cover the spec round-trip and every rejection, the
sweep (inclusive cutoff, string/missing/future values untouched, second
sweep a no-op) and its survival of a reopen, and two TTL indexes over
one collection. e2e4.js exercises it through the Node driver.
Cleanup pass over the secondary-index feature.
add_doc is now the one entry-commit path. create_index and
build_all_indexes each hand-rolled build -> check_unique -> reserve ->
insert, and had already drifted on whether multikey is set before or
after the unique check; add_doc gained an enforce_unique flag so the
rebuild path keeps its tolerate-and-warn behavior. reserve_for and
insert_entries are now the only way the engine touches Index.entries.
One definition each for: prefix comparison and the prefix binary
searches (prefix_order + std.sort), the cartesian-product odometer
(advance_choice), the spec pair list (write_spec builds on spec_pairs,
so the log format and the listIndexes reply share one schema), the _id
clause parser (plan_id reuses analyze_clause), key-pattern direction
(index.descending, which desc_dir already disagreed with on non-numeric
values), option truthiness (query.truthy), the E11000 message, and
index-removal-by-name (Collection.find_index/remove_index). Key-pattern
matching moved out of the dispatcher into index.find_by_key_pattern.
Dead or redundant: ParallelArraysError, the unread `dropped` counter,
insert_entries' discarded gpa, a third pass computing multikey, the
has_id/is_id_index flag pair, Plan.key_len (always lookup_keys[0].len,
now a method), first_match_consumed (now stages = stages[1..]).
Cheaper hot paths: remove_id compacts in one pass instead of an
orderedRemove per hit; Plan.search skips the sort/dedupe when neither
multikey nor multiple lookup keys can produce a repeat; the _id fast
path reuses one scratch key buffer (bson.write_serialized_value); the
plan loop uses the bound collection instead of re-resolving it through
two hash lookups per candidate.
Behavior is unchanged except that dropping plan_id's fixed 16-clause
buffer enables the _id fast path on filters that previously exceeded it.
Adds the three driver commands, parameterized E11000 messages (engine
dup_index carries the index name into writeErrors), the scan_matching
planner wiring (_id fast path → index plan → scan) and the first-$match
aggregate pushdown. The equivalence test (mixed-type corpus x 27 filters,
non-sparse and sparse indexes) drove out three real bugs: the two-bound
range under-approximation on multikey indexes (fall back to scan), a
dangling single-value option array in the planner, and update-time
unique violations now reporting writeErrors instead of corrupting state.
Collection gains an indexes list; evict_doc removes entries at the single
document-death chokepoint; upsert does build → check → reserve → log →
evict → publish so entry insertion after the append is infallible and a
rejected unique write never reaches the log. Replay registers empty
indexes from create/drop records (types 3/4) and Engine.open rebuilds them
from live docs. compact re-emits index-create records. Engine gains
create_index/drop_index and dup_index for E11000 naming.
Adds src/index.zig with the full secondary-index machinery: entry
generation mirroring field_matches (array value + elements), BSON-order
sorted entries with binary search, compound prefix and range lookups,
unique/sparse options, the query planner (longest equality/$in run +
optional range, $in cartesian cap, sparse/null bail), and the _id_ fast
path guarded against serialization-ambiguous values (numbers, strings,
symbols, codes, opaque payloads).
query.collect_values is now pub so entry generation can mirror it exactly.
storage.zig gains record_type_index_create/drop; lib.zig exports index.
The Node driver sends {field: /re/} as a BSON regex element (type 0x0B),
which the matcher previously only handled via the $regex operator form;
and dot paths with numeric segments (tags.0) were ignored because array
descent only recursed into embedded docs. Both are part of standard
MongoDB query semantics and were caught by the driver e2e suite.
server: unbounded Io async limit so the accept loop never wedges, and
treat header-read failures (client RST on pool teardown) as clean
disconnects. With the default cpu_count-1 limit, groupAsync's eager
fallback ran connection handlers inline on the accept-loop fiber once
that many connections were alive, stalling accept() and timing out
handshakes for further clients.
Add tests/e2e/: official driver CRUD, concurrency, and kill -9 recovery
suites (29 + 2 + 3 checks), plus unit tests for the query fixes.
The original dispatch restructure placed the unlock defers inside switch
prongs, where Zig runs them when the prong block exits — immediately after
acquisition. All commands therefore ran with no lock at all, which the new
stress test caught deterministically (two writers inside Engine.insert at
once). Lock acquisition now happens in the prongs and the command body runs
via dispatch_impl, so the defers (still prong-scoped) release the lock only
after the command finishes.
