PLAN amendment A3. The B+tree leaf had nowhere to put a document's slab
offset -- `Slot.extra` is the payload length for a leaf and the child node id
for an internal separator -- which is what blocks the `_id_` tree from becoming
the primary lookup once the docs hashmap goes away.
A leaf record is now `key ++ offset_le`, so `extra` is always 8 and every
byte-accounting site (fits, record_cost, slot_cost, balanced_cut,
repack_keep_prefix) is untouched. Records get *smaller*: an ObjectId `_id_`
record goes from 26 bytes to 21.
`Entry.id` is deleted rather than re-owned. Every entry one document
contributes shares one document, so which document it is belongs on the call
that commits the entries -- which also makes it impossible to confuse the
offset a replace is removing with the one it is inserting. The old field
aliased the docs map's key and was only safe because removal happened at the
one chokepoint where a document dies; that constraint is gone.
Done for secondary indexes too, not just `_id_`. That deletes the per-candidate
`coll.docs.get(id)` in scan_sorted outright rather than replacing it with an
`_id_` descent, and it is free on the write path because a replace already
removes and reinserts every entry in every index.
Consequences worth knowing:
- lookup_eq/lookup_range/Plan.search yield u64. Those are values, immune to the
tree mutation that invalidated the id slices they used to hand back -- which
is why ttl_sweep_coll can drop the dupe-and-free dance it needed to survive
`remove` freeing the key its entries pointed at.
- One safety net is gone. A stale entry used to be swallowed by
`docs.get(id) orelse continue`; now it resolves to superseded-but-parseable
bytes the re-applied filter might accept. That trades an invisible
under-approximation for a visible wrong answer, which is the better failure
to have, but it is a trade.
- A checkpoint may never renumber slab offsets (already recorded in PLAN §4):
every index leaf now holds a physical one.
`zig build fuzz` earned its keep immediately -- it caught the API break in all
four B+tree harnesses, which `zig build test` cannot see.
Benchmarks A/B'd at 256m on one harness, before and after: all rows flat.
updateMany and deleteOne+insertOne first looked 10-13% slower, which three
repeat runs showed to be single-sample noise (0.70/0.71/0.70 against 0.70).
Wrap signatures and long expressions to the 100-column limit and make every
file zig fmt clean. Semantics-preserving throughout: ignoring whitespace and
the trailing commas that wrapping introduces, every file here is byte-identical
to its predecessor, and the one apparent exception is a warning string split
with `++`, which concatenates at comptime to the same bytes.
src/index.zig and src/commands.zig are reformatted in the commits that follow,
because their reformat is interleaved with in-flight changes to them and
separating the two would need the reformat re-derived rather than moved.
Documents live as canonical BSON bytes in a segmented per-collection slab
(fixed 8 MiB segments keep capacity slack under one segment); the docs map
holds flat offsets that stay valid across segment growth, and removed
documents leave garbage bytes until compaction rewrites. The per-document
ArenaAllocator and its second full Pair-tree copy are gone.
The matcher walks the stored bytes directly, skipping by length any field
the filter does not name (a new bson byte-walker: element_key, skip_value,
read_value with borrowed leaves, get_at, and a borrowed spine parse). The
byte matcher is differential-tested against the tree matcher on a corpus
and shares its operator logic. Stored documents are never materialized on
the scan path or in aggregate $match; $group reads group keys and sums
straight off the bytes. Sort, projection, findAndModify, updates and
index entry generation use a borrowed spine into the slab (or the byte
collector, which also replaced collect_values in build_entries). The
compaction threshold now counts uncompressed data volume, since a
compressed log would otherwise never trigger.
Measured (tests/e2e/results/phase5.txt): server RSS 1979 -> 539 MB (2.4x
smaller than MongoDB; phase1 baseline 2.0 GB), range-scan 22.5 -> ~12 ms
(parity, best run faster than MongoDB), proj 4.1 -> 3.4 ms, createIndex
parity. Verified: unit suite in all three modes with zero leaks, the
crash pair, e2e6, and the stress/spill programs.
Replace Index.entries (one sorted array) with a B+tree so writes into an
already-built index stop being quadratic. Nodes are fixed 4 KiB slotted
pages in a flat u32-addressed ArrayListUnmanaged(Node); records longer
than a quarter page spill to an append-only overflow slab (BSON strings
reach 16 MB). Leaves are doubly linked for ordered iteration; the flat
node array stays one contiguous byte range for a later checkpoint.
Insertion descends by separator and splits leaves/internals upward,
promoting keys via a stable copy (a nested split can otherwise clobber
the promoted-key scratch). Deletion does not rebalance: emptied leaves
are unlinked and dropped from their parent, internal nodes may carry one
child, and dead pages are abandoned in place (node memory peaks at the
tree's peak size, exactly what the old array's capacity did). Lookups
are lower-bound seeks plus leaf-chain band scans, so equal keys may
span leaves freely. Bulk build (append_doc_entries + finish_bulk) sorts
a staging array and packs leaves bottom-up. reserve_for now takes the
built entries and reserves exact overflow bytes plus a worst-case node
count, keeping insert_entries infallible after the log append.
db.zig: TTL sweep now seeks the minimum-datetime encoded key and walks
the contiguous datetime band, stopping at the cutoff or type change.
Measured (tests/e2e/results/phase2.txt): updateMany 17.3 -> 1.8 ms
(2.8x slower than MongoDB -> 3.7x faster), createIndex 62 -> 51 ms.
Verified: unit suite ReleaseFast/ReleaseSafe/Debug (incl. the existing
lookup_range and remove_doc differentials, plus a new incremental
insert/remove differential against a brute-force model), the crash pair,
e2e3/e2e4/e2e6, and dev stress tests for depth-2 splits, full drains,
and spilled records through internal levels.
