An accessor over the command envelope, next to `db_name` and the same shape:
called from dispatch, never from `parse`, because a malformed session id is a
command that gets an error reply, not a connection that gets torn down. It
returns the 16 bytes or names what is wrong; the codes stay in the command
layer, where they were measured.
The tolerated fields were measured against mongod 8.3.7 rather than recalled,
and the measurement contradicted the assumption this was designed on. The
design said unknown fields inside `lsid` would be tolerated, on the reasoning
that the server tolerates unknown fields everywhere and pinpoint strictness
would be inconsistent. mongod answers IDLUnknownField (40415) -- it is strict
here and the reasoning was simply wrong. It also accepts `uid`, the hash of
the credentials owning the session, which a driver starts sending the moment
authentication is on; rejecting that would have broken every command in M7,
and the test says so where a future reader will meet it.
`txnNumber` and `txnUUID` inside `lsid` are refused. They are not the
retryable-write `txnNumber` that sits outside it: together they name an
*internal* session, one that runs a transaction on another session's behalf.
mongod refuses them on a standalone too.
Also `bson.Value.type_name`, which is mongod's name for a type rather than
Zig's -- a TypeMismatch message quotes it, and a driver that matches on the
text is matching on these.
170/170 unit tests.
Wrap signatures and long expressions to the 100-column limit and make every
file zig fmt clean. Semantics-preserving throughout: ignoring whitespace and
the trailing commas that wrapping introduces, every file here is byte-identical
to its predecessor, and the one apparent exception is a warning string split
with `++`, which concatenates at comptime to the same bytes.
src/index.zig and src/commands.zig are reformatted in the commits that follow,
because their reformat is interleaved with in-flight changes to them and
separating the two would need the reformat re-derived rather than moved.
Documents live as canonical BSON bytes in a segmented per-collection slab
(fixed 8 MiB segments keep capacity slack under one segment); the docs map
holds flat offsets that stay valid across segment growth, and removed
documents leave garbage bytes until compaction rewrites. The per-document
ArenaAllocator and its second full Pair-tree copy are gone.
The matcher walks the stored bytes directly, skipping by length any field
the filter does not name (a new bson byte-walker: element_key, skip_value,
read_value with borrowed leaves, get_at, and a borrowed spine parse). The
byte matcher is differential-tested against the tree matcher on a corpus
and shares its operator logic. Stored documents are never materialized on
the scan path or in aggregate $match; $group reads group keys and sums
straight off the bytes. Sort, projection, findAndModify, updates and
index entry generation use a borrowed spine into the slab (or the byte
collector, which also replaced collect_values in build_entries). The
compaction threshold now counts uncompressed data volume, since a
compressed log would otherwise never trigger.
Measured (tests/e2e/results/phase5.txt): server RSS 1979 -> 539 MB (2.4x
smaller than MongoDB; phase1 baseline 2.0 GB), range-scan 22.5 -> ~12 ms
(parity, best run faster than MongoDB), proj 4.1 -> 3.4 ms, createIndex
parity. Verified: unit suite in all three modes with zero leaks, the
crash pair, e2e6, and the stress/spill programs.
Entry.key becomes the order-preserving byte encoding of the indexed
values, concatenated column by column, instead of a slice of Values.
Comparing two entries is now a memcmp.
The old representation allocated one Value slice per entry, and each
Value in it pointed into a different document's arena -- so a binary
search over the entry array was a chain of pointer chases across the
heap, and every comparison walked the key component by component
dispatching on BSON type. Byte keys make the comparison contiguous and
type-free, and the key no longer aliases the document at all.
createIndex over 65,536 documents:
{k: 1} 56ms -> 44ms
{s: 1} unique 53ms -> 31ms
{p: 1, k: -1} 54ms -> 37ms
(both already down from ~650ms before the bulk build)
The search API still takes Values and encodes at the call site: lookups
happen per query, not per document, so there is nothing to gain from
pushing the encoding out to callers, and Plan keeps its current shape.
Prefix search compares raw byte prefixes, which is sound because every
column encoding is self-delimiting -- a prefix of an encoded key is
exactly the encoding of its leading columns. For the same reason a
complete column encoding can never be a proper prefix of another, so
finish_bulk's duplicate test is now a plain byte equality.
The TTL sweep read entry keys as Values to find datetimes. It now uses
bson.encoded_leading_datetime, which checks the column's tag and decodes
eight bytes rather than the whole key. Still a linear walk for the reason
the existing comment gives.
Key direction is deliberately still not applied to the encoding.
Complementing descending columns would let a sort read the array
forwards, but nothing exploits that yet, and doing it now would change
the array's order for no gain. It belongs with the sort-aware planner.
remove_id is still a linear scan and insertion still memmoves the tail:
those are the tree's job, not this change's.
Verified: 77 unit tests under ReleaseFast and ReleaseSafe, e2e
29/16/17/3/2, the crash pair, e2e6 72/72, and the randomized
lookup_range test that checks bounds against a brute-force filter.
Encodes a Value so that std.mem.order over the bytes reproduces
bson.compare exactly. This is the foundation for the encoded-key index:
it lets an index binary-search, range-scan and eventually be stored as
raw bytes, instead of carrying Value trees whose every comparison chases
pointers into a different document's arena.
Layout is [rank + 1] then a self-delimiting payload; the +1 keeps 0x00
out of the tag space so it can terminate variable-length payloads.
The parts that are easy to get wrong, and why they are the way they are:
Numbers encode the f128 that compare already widens int32, int64 and
double to -- exactly, for all three. So int32 1, int64 1 and double 1.0
produce identical bytes, which is the cross-type equality that numeric
index lookups need, and is precisely what value_fast_path_safe exists
today to work around. Negatives are bit-inverted and positives get the
sign bit set, making the IEEE order lexicographic. -0.0 normalizes to
+0.0 (they compare equal) and every NaN encodes as all-ones (compare
makes NaN greatest and all NaNs equal).
Byte strings escape 0x00 as 00 FF and terminate with 00 00. A BSON
string may contain NUL, so a bare terminator would be ambiguous;
escaping fixes ordering at the same time, since a real NUL then sorts
above the terminator and any byte >= 01 does too. "Shorter is less"
falls out to match std.mem.order, which also gives documents and arrays
their length tie-break for free.
Binary length-prefixes because compare_binary orders by length first,
but opaque_val escapes instead: compare ignores its kind and orders the
data lexicographically, not by length.
Correctness rests entirely on the order equivalence, so it is checked
exhaustively rather than by example: every ordered pair of a corpus
spanning all fifteen ranks and their boundaries (numeric cross-type and
sign, NaN, both zeros, infinities, embedded NULs, prefix relationships,
empty and nested documents and arrays, binary subtypes) is compared both
ways. A second test concatenates two-column keys and checks they
reproduce component-wise order, which is what makes compound keys and
prefix search sound. Verified both fail when escaping is dropped, when
-0.0 is not normalized, when the binary length prefix is wrong, and when
NaN stops being greatest.
Nothing uses the encoding yet; the index still holds Value keys.
Cleanup pass over the secondary-index feature.
add_doc is now the one entry-commit path. create_index and
build_all_indexes each hand-rolled build -> check_unique -> reserve ->
insert, and had already drifted on whether multikey is set before or
after the unique check; add_doc gained an enforce_unique flag so the
rebuild path keeps its tolerate-and-warn behavior. reserve_for and
insert_entries are now the only way the engine touches Index.entries.
One definition each for: prefix comparison and the prefix binary
searches (prefix_order + std.sort), the cartesian-product odometer
(advance_choice), the spec pair list (write_spec builds on spec_pairs,
so the log format and the listIndexes reply share one schema), the _id
clause parser (plan_id reuses analyze_clause), key-pattern direction
(index.descending, which desc_dir already disagreed with on non-numeric
values), option truthiness (query.truthy), the E11000 message, and
index-removal-by-name (Collection.find_index/remove_index). Key-pattern
matching moved out of the dispatcher into index.find_by_key_pattern.
Dead or redundant: ParallelArraysError, the unread `dropped` counter,
insert_entries' discarded gpa, a third pass computing multikey, the
has_id/is_id_index flag pair, Plan.key_len (always lookup_keys[0].len,
now a method), first_match_consumed (now stages = stages[1..]).
Cheaper hot paths: remove_id compacts in one pass instead of an
orderedRemove per hit; Plan.search skips the sort/dedupe when neither
multikey nor multiple lookup keys can produce a repeat; the _id fast
path reuses one scratch key buffer (bson.write_serialized_value); the
plan loop uses the bound collection instead of re-resolving it through
two hash lookups per candidate.
Behavior is unchanged except that dropping plan_id's fixed 16-clause
buffer enables the _id fast path on filters that previously exceeded it.
Adds src/index.zig with the full secondary-index machinery: entry
generation mirroring field_matches (array value + elements), BSON-order
sorted entries with binary search, compound prefix and range lookups,
unique/sparse options, the query planner (longest equality/$in run +
optional range, $in cartesian cap, sparse/null bail), and the _id_ fast
path guarded against serialization-ambiguous values (numbers, strings,
symbols, codes, opaque payloads).
query.collect_values is now pub so entry generation can mirror it exactly.
storage.zig gains record_type_index_create/drop; lib.zig exports index.
