index: hold encoded byte keys instead of Value slices

Entry.key becomes the order-preserving byte encoding of the indexed
values, concatenated column by column, instead of a slice of Values.
Comparing two entries is now a memcmp.

The old representation allocated one Value slice per entry, and each
Value in it pointed into a different document's arena -- so a binary
search over the entry array was a chain of pointer chases across the
heap, and every comparison walked the key component by component
dispatching on BSON type. Byte keys make the comparison contiguous and
type-free, and the key no longer aliases the document at all.

  createIndex over 65,536 documents:
    {k: 1}          56ms -> 44ms
    {s: 1} unique   53ms -> 31ms
    {p: 1, k: -1}   54ms -> 37ms
  (both already down from ~650ms before the bulk build)

The search API still takes Values and encodes at the call site: lookups
happen per query, not per document, so there is nothing to gain from
pushing the encoding out to callers, and Plan keeps its current shape.

Prefix search compares raw byte prefixes, which is sound because every
column encoding is self-delimiting -- a prefix of an encoded key is
exactly the encoding of its leading columns. For the same reason a
complete column encoding can never be a proper prefix of another, so
finish_bulk's duplicate test is now a plain byte equality.

The TTL sweep read entry keys as Values to find datetimes. It now uses
bson.encoded_leading_datetime, which checks the column's tag and decodes
eight bytes rather than the whole key. Still a linear walk for the reason
the existing comment gives.

Key direction is deliberately still not applied to the encoding.
Complementing descending columns would let a sort read the array
forwards, but nothing exploits that yet, and doing it now would change
the array's order for no gain. It belongs with the sort-aware planner.

remove_id is still a linear scan and insertion still memmoves the tail:
those are the tree's job, not this change's.

Verified: 77 unit tests under ReleaseFast and ReleaseSafe, e2e
29/16/17/3/2, the crash pair, e2e6 72/72, and the randomized
lookup_range test that checks bounds against a brute-force filter.
This commit is contained in:
2026-08-02 19:28:40 +03:00
parent ea1f0f09cf
commit 6feacc21cd
3 changed files with 83 additions and 53 deletions

View File

@@ -426,12 +426,15 @@ pub const Engine = struct {
const ttl = ix.ttl orelse continue;
const cutoff: i128 = @as(i128, now_ms) - @as(i128, ttl) * 1000;
for (ix.entries.items) |e| {
// The type test cannot be a range lookup: bson
// compare order ranks datetime above null, numbers
// and strings, so a datetime upper bound would also
// select every value of a lesser type.
if (e.key[0] != .datetime) continue;
if (@as(i128, e.key[0].datetime) > cutoff) continue;
// Still a linear walk: the type test cannot be a
// one-sided range lookup, because bson compare order
// ranks datetime above null, numbers and strings, so
// a datetime upper bound would also select every
// value of a lesser type. (Datetimes are contiguous
// in that order, so a two-sided band lookup would
// work — that comes with the tree.)
const ms = bson.encoded_leading_datetime(e.key) orelse continue;
if (@as(i128, ms) > cutoff) continue;
try ids.append(self.gpa, try self.gpa.dupe(u8, e.id));
}
}