M3's last row, step 4 of the index review's order. `{a: "hashed"}` was
refused with "invalid index spec" -- the right answer with the wrong code,
and the half of the row the review called honestly missing.
A hashed component is stored as a tag byte plus a 64-bit hash of the
value's *ordinary* encoded bytes. Hashing the encoding rather than the
value is what makes `{a: 5}` and `{a: 5.0}` land on the same entry for
free: `bson.encode_key` already normalizes every numeric type through
f128, because two values that compare `.eq` have to encode identically for
the tree to be a memcmp. One `encode_component` does it for entry
generation and both lookup paths, so the two sides cannot disagree -- a
component hashed on the way in and not on the way out would simply never
find anything.
Collisions are harmless, because this file's governing invariant is that
an index only generates candidates and the full filter is re-applied to
every one. The single place that would not survive one is uniqueness,
which is why `unique` is refused (16764) rather than approximated.
The planner is the mirror of the partial rule and just as conservative:
equality only. A range or a sort over a hashed component would read a band
of leaves ordered by hash, which is an arbitrary set of values, so both
are declined and the query scans. That is what leaves `find({a: {$gte:
5}})` and `find({}, {sort: {a: 1}})` correct.
The catalog needs no new field. `write_index_catalog` has always written
one byte per component and that byte has only ever held 0 or 1, so a third
value costs no format change and `catalog_version` stays 1. That is a
departure from the review, which guessed at a sixth flags bit: hashed
belongs to a *component*, and a compound index may hold one beside range
ones.
Measured on mongod 8.3.7 rather than recalled, and three of the five
answers were not what the corpus source assumed:
two hashed components 31303, codeName Location31303
unique on a hashed index 16764, codeName Location16764
an unknown plugin string 67, codeName CannotCreateIndex
an array at the path 16766 -- a *writeError* beside `ok: 1` on an
insert or update, and a command error from
createIndexes over data that already holds one
an array through a path refused for a *one-element* array too, which
is why `array_on_path` walks the path instead
of counting the values at it
tests/spec/indexes/hashed.json goes 0/18 -> 17/18. The one that remains
is not about hashed indexes: `find({a: null})` has to match a document
with no `a`, and this server matches only an explicit null -- with or
without an index. Next commit.
The last structures move onto the pager, so the whole engine's storage is now
one mapped file plus the WAL.
Node ids are deliberately *not* page numbers. PLAN amendment A1 explains why:
`Node.parent`, `next`, `prev` and an internal slot's `extra` are back-pointers
by id, so copy-on-write moving a page would force every node referring to it to
move as well -- COWing one leaf cascades through the leaf level, one internal
node through its whole subtree. An in-RAM id->page table makes the table slot
the single owner of a page number, so COW has exactly one pointer to fix. It
costs one dependent load per node access and 4 bytes per node, about 5.6 MB at
100M documents, against the 64-100 bytes *per document* this milestone removes.
The overflow slab becomes extents too, so `Slot.off` for a spilled record is an
absolute file offset -- the same change documents went through.
--
Two bugs, both found by measuring rather than by reading, and both worth
recording because the second one would have been invisible until the churn gate.
The reservation was a tail mark, and it cannot be: an upsert reserves tree pages
for every index *and* slab room for the document, all before one log append. The
second reserver overwrote the first one's promise and the first one's allocation
then asserted. Caught on a 512 MB load by the tripwire added in the
`reserve_for` commit, which is the entire reason that assert exists. It is a
count now, and the multi-consumer ordering is pinned by a test.
And a reservation was never released. It is scoped to one write -- taken before
the log append so the publish cannot fail -- but a tree reservation covers the
worst case of several splits while a typical insert causes none, so the promise
accumulated by a handful of pages per write and dragged the file up with it. The
data file was **1.89 GB for 512 MB of documents**; releasing the unclaimed
promise at the end of each write brings it to 551 MB, or 1.08x, which is the
extent slack and the node pages.
--
Measured on one harness, 512 MB / 16 KB docs, against the in-RAM engine this
replaces:
bulk insert throughput 742.6 MB/s -> 736.4 MB/s
insertOne (sequential) 0.20 ms -> 0.22 ms
createIndex({k: 1}) 26.8 ms -> 16.5 ms
countDocuments({}) 2.1 ms -> 1.1 ms
findOne({k: 500}) indexed 0.75 ms -> 0.56 ms
find({p: range}).count() 6.6 ms -> 4.6 ms
aggregate $group by k 5.8 ms -> 3.8 ms
updateMany({k: 7}, {$inc}) 1.2 ms -> 1.0 ms
Reads gain from one contiguous mapping; the two write rows are within noise of
flat. RSS is still unchanged and still cannot improve, for the reason given in
the previous commit: every open replays the whole log and rebuilds everything.
The dev harnesses each open their own data file now. `zig build fuzz` caught all
four of them, again.
PLAN amendment A3. The B+tree leaf had nowhere to put a document's slab
offset -- `Slot.extra` is the payload length for a leaf and the child node id
for an internal separator -- which is what blocks the `_id_` tree from becoming
the primary lookup once the docs hashmap goes away.
A leaf record is now `key ++ offset_le`, so `extra` is always 8 and every
byte-accounting site (fits, record_cost, slot_cost, balanced_cut,
repack_keep_prefix) is untouched. Records get *smaller*: an ObjectId `_id_`
record goes from 26 bytes to 21.
`Entry.id` is deleted rather than re-owned. Every entry one document
contributes shares one document, so which document it is belongs on the call
that commits the entries -- which also makes it impossible to confuse the
offset a replace is removing with the one it is inserting. The old field
aliased the docs map's key and was only safe because removal happened at the
one chokepoint where a document dies; that constraint is gone.
Done for secondary indexes too, not just `_id_`. That deletes the per-candidate
`coll.docs.get(id)` in scan_sorted outright rather than replacing it with an
`_id_` descent, and it is free on the write path because a replace already
removes and reinserts every entry in every index.
Consequences worth knowing:
- lookup_eq/lookup_range/Plan.search yield u64. Those are values, immune to the
tree mutation that invalidated the id slices they used to hand back -- which
is why ttl_sweep_coll can drop the dupe-and-free dance it needed to survive
`remove` freeing the key its entries pointed at.
- One safety net is gone. A stale entry used to be swallowed by
`docs.get(id) orelse continue`; now it resolves to superseded-but-parseable
bytes the re-applied filter might accept. That trades an invisible
under-approximation for a visible wrong answer, which is the better failure
to have, but it is a trade.
- A checkpoint may never renumber slab offsets (already recorded in PLAN §4):
every index leaf now holds a physical one.
`zig build fuzz` earned its keep immediately -- it caught the API break in all
four B+tree harnesses, which `zig build test` cannot see.
Benchmarks A/B'd at 256m on one harness, before and after: all rows flat.
updateMany and deleteOne+insertOne first looked 10-13% slower, which three
repeat runs showed to be single-sample noise (0.70/0.71/0.70 against 0.70).
Wrap signatures and long expressions to the 100-column limit and make every
file zig fmt clean. Semantics-preserving throughout: ignoring whitespace and
the trailing commas that wrapping introduces, every file here is byte-identical
to its predecessor, and the one apparent exception is a warning string split
with `++`, which concatenates at comptime to the same bytes.
src/index.zig and src/commands.zig are reformatted in the commits that follow,
because their reformat is interleaved with in-flight changes to them and
separating the two would need the reformat re-derived rather than moved.
Documents live as canonical BSON bytes in a segmented per-collection slab
(fixed 8 MiB segments keep capacity slack under one segment); the docs map
holds flat offsets that stay valid across segment growth, and removed
documents leave garbage bytes until compaction rewrites. The per-document
ArenaAllocator and its second full Pair-tree copy are gone.
The matcher walks the stored bytes directly, skipping by length any field
the filter does not name (a new bson byte-walker: element_key, skip_value,
read_value with borrowed leaves, get_at, and a borrowed spine parse). The
byte matcher is differential-tested against the tree matcher on a corpus
and shares its operator logic. Stored documents are never materialized on
the scan path or in aggregate $match; $group reads group keys and sums
straight off the bytes. Sort, projection, findAndModify, updates and
index entry generation use a borrowed spine into the slab (or the byte
collector, which also replaced collect_values in build_entries). The
compaction threshold now counts uncompressed data volume, since a
compressed log would otherwise never trigger.
Measured (tests/e2e/results/phase5.txt): server RSS 1979 -> 539 MB (2.4x
smaller than MongoDB; phase1 baseline 2.0 GB), range-scan 22.5 -> ~12 ms
(parity, best run faster than MongoDB), proj 4.1 -> 3.4 ms, createIndex
parity. Verified: unit suite in all three modes with zero leaks, the
crash pair, e2e6, and the stress/spill programs.
Replace Index.entries (one sorted array) with a B+tree so writes into an
already-built index stop being quadratic. Nodes are fixed 4 KiB slotted
pages in a flat u32-addressed ArrayListUnmanaged(Node); records longer
than a quarter page spill to an append-only overflow slab (BSON strings
reach 16 MB). Leaves are doubly linked for ordered iteration; the flat
node array stays one contiguous byte range for a later checkpoint.
Insertion descends by separator and splits leaves/internals upward,
promoting keys via a stable copy (a nested split can otherwise clobber
the promoted-key scratch). Deletion does not rebalance: emptied leaves
are unlinked and dropped from their parent, internal nodes may carry one
child, and dead pages are abandoned in place (node memory peaks at the
tree's peak size, exactly what the old array's capacity did). Lookups
are lower-bound seeks plus leaf-chain band scans, so equal keys may
span leaves freely. Bulk build (append_doc_entries + finish_bulk) sorts
a staging array and packs leaves bottom-up. reserve_for now takes the
built entries and reserves exact overflow bytes plus a worst-case node
count, keeping insert_entries infallible after the log append.
db.zig: TTL sweep now seeks the minimum-datetime encoded key and walks
the contiguous datetime band, stopping at the cutoff or type change.
Measured (tests/e2e/results/phase2.txt): updateMany 17.3 -> 1.8 ms
(2.8x slower than MongoDB -> 3.7x faster), createIndex 62 -> 51 ms.
Verified: unit suite ReleaseFast/ReleaseSafe/Debug (incl. the existing
lookup_range and remove_doc differentials, plus a new incremental
insert/remove differential against a brute-force model), the crash pair,
e2e3/e2e4/e2e6, and dev stress tests for depth-2 splits, full drains,
and spilled records through internal levels.
