dev/MultiforaDB
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

index: hold encoded byte keys instead of Value slices

Browse Source 
Entry.key becomes the order-preserving byte encoding of the indexed
values, concatenated column by column, instead of a slice of Values.
Comparing two entries is now a memcmp.

The old representation allocated one Value slice per entry, and each
Value in it pointed into a different document's arena -- so a binary
search over the entry array was a chain of pointer chases across the
heap, and every comparison walked the key component by component
dispatching on BSON type. Byte keys make the comparison contiguous and
type-free, and the key no longer aliases the document at all.

  createIndex over 65,536 documents:
    {k: 1}          56ms -> 44ms
    {s: 1} unique   53ms -> 31ms
    {p: 1, k: -1}   54ms -> 37ms
  (both already down from ~650ms before the bulk build)

The search API still takes Values and encodes at the call site: lookups
happen per query, not per document, so there is nothing to gain from
pushing the encoding out to callers, and Plan keeps its current shape.

Prefix search compares raw byte prefixes, which is sound because every
column encoding is self-delimiting -- a prefix of an encoded key is
exactly the encoding of its leading columns. For the same reason a
complete column encoding can never be a proper prefix of another, so
finish_bulk's duplicate test is now a plain byte equality.

The TTL sweep read entry keys as Values to find datetimes. It now uses
bson.encoded_leading_datetime, which checks the column's tag and decodes
eight bytes rather than the whole key. Still a linear walk for the reason
the existing comment gives.

Key direction is deliberately still not applied to the encoding.
Complementing descending columns would let a sort read the array
forwards, but nothing exploits that yet, and doing it now would change
the array's order for no gain. It belongs with the sort-aware planner.

remove_id is still a linear scan and insertion still memmoves the tail:
those are the tree's job, not this change's.

Verified: 77 unit tests under ReleaseFast and ReleaseSafe, e2e
29/16/17/3/2, the crash pair, e2e6 72/72, and the randomized
lookup_range test that checks bounds against a brute-force filter.
This commit is contained in:
Alexey Shakhmatov
2026-08-02 19:28:40 +03:00
parent ea1f0f09cf
commit 6feacc21cd
3 changed files with 83 additions and 53 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
src/bson.zig
View File

@@ -617,8 +617,9 @@ pub fn encode_key(v: Value, gpa: std.mem.Allocator, out: *std.ArrayListUnmanaged
// Flip the sign bit so the two's-complement order becomes unsigned
// byte order.
.datetime => |ms| {
std.debug.assert(rank(v) + 1 == encoded_datetime_tag);
var buf: [8]u8 = undefined;
std.mem.writeInt(u64, &buf, @as(u64, @bitCast(ms)) ^ (1 << 63), .big);
std.mem.writeInt(u64, &buf, @as(u64, @bitCast(ms)) ^ (@as(u64, 1) << 63), .big);
try out.appendSlice(gpa, &buf);
},
.timestamp => |ts| {
@@ -638,6 +639,18 @@ pub fn encode_key(v: Value, gpa: std.mem.Allocator, out: *std.ArrayListUnmanaged
}
}
/// The tag `encode_key` writes for a datetime.
pub const encoded_datetime_tag: u8 = 9 + 1;
/// The datetime in an encoded key's first column, or null when that column
/// holds anything else. Lets a TTL sweep read entry keys without decoding
/// them back into Values.
pub fn encoded_leading_datetime(key: []const u8) ?i64 {
if (key.len < 1 + 8 or key[0] != encoded_datetime_tag) return null;
const biased = std.mem.readInt(u64, key[1..9], .big);
return @bitCast(biased ^ (@as(u64, 1) << 63));
}
/// Byte string, terminated so it stays self-delimiting, with the terminator
/// ordering below any content.
///

15
src/db.zig
View File

@@ -426,12 +426,15 @@ pub const Engine = struct {
const ttl = ix.ttl orelse continue;
const cutoff: i128 = @as(i128, now_ms) - @as(i128, ttl) * 1000;
for (ix.entries.items) |e| {
// The type test cannot be a range lookup: bson
// compare order ranks datetime above null, numbers
// and strings, so a datetime upper bound would also
// select every value of a lesser type.
if (e.key[0] != .datetime) continue;
if (@as(i128, e.key[0].datetime) > cutoff) continue;
// Still a linear walk: the type test cannot be a
// one-sided range lookup, because bson compare order
// ranks datetime above null, numbers and strings, so
// a datetime upper bound would also select every
// value of a lesser type. (Datetimes are contiguous
// in that order, so a two-sided band lookup would
// work — that comes with the tree.)
const ms = bson.encoded_leading_datetime(e.key) orelse continue;
if (@as(i128, ms) > cutoff) continue;
try ids.append(self.gpa, try self.gpa.dupe(u8, e.id));
}
}

102
src/index.zig
View File

@@ -20,11 +20,12 @@
//! - Entry insertion is infallible after the log append (capacity is
//! reserved first), so a document can never be live but unindexed.
//!
//! Entries alias: `Entry.key` values point into the stored document's arena
//! and `Entry.id` aliases the docs map key; only the `[]bson.Value` slice
//! itself is owned (freed on entry removal). Index removal happens at the
//! top of evict_doc (src/db.zig) — the single chokepoint where a document
//! dies — so the aliasing is safe by construction.
//! `Entry.key` is an owned, order-preserving byte encoding of the indexed
//! values (see bson.encode_key), so comparing two entries is a memcmp
//! rather than a walk over Values that each live in a different document's
//! arena. `Entry.id` still aliases the docs map key; index removal happens
//! at the top of evict_doc (src/db.zig) — the single chokepoint where a
//! document dies — so that aliasing is safe by construction.
const std = @import("std");
const bson = @import("bson.zig");
@@ -43,10 +44,11 @@ pub const IndexKey = struct {
descending: bool,
};
/// One index entry. `key` is a gpa-owned slice of Values aliasing the
/// stored document's arena; `id` aliases the docs map key.
/// One index entry. `key` is the gpa-owned encoded form of this document's
/// values for the index's paths, concatenated column by column; `id`
/// aliases the docs map key.
pub const Entry = struct {
key: []const bson.Value,
key: []const u8,
id: []const u8,
};
@@ -159,10 +161,15 @@ pub const Index = struct {
for (0..nkeys) |ci| limits[ci] = per_path.items[ci].items.len;
var choice: [max_index_keys]usize = undefined;
@memset(choice[0..nkeys], 0);
// One reused buffer; each finished key is copied out to its own
// exact-sized allocation.
var enc: std.ArrayListUnmanaged(u8) = .empty;
defer enc.deinit(gpa);
while (true) {
const key = try gpa.alloc(bson.Value, nkeys);
enc.clearRetainingCapacity();
for (0..nkeys) |ci| try bson.encode_key(per_path.items[ci].items[choice[ci]], gpa, &enc);
const key = try gpa.dupe(u8, enc.items);
errdefer gpa.free(key);
for (0..nkeys) |ci| key[ci] = per_path.items[ci].items[choice[ci]];
try out.append(gpa, .{ .key = key, .id = id });
if (!advance_choice(choice[0..nkeys], limits[0..nkeys])) break;
}
@@ -264,10 +271,8 @@ pub const Index = struct {
var duplicate = false;
for (self.entries.items[1..], 0..) |cur, prev_i| {
const prev = self.entries.items[prev_i];
// Every key in one index has the same component count, so a
// prefix comparison over the previous key is a full key
// comparison. Sorting puts equal keys next to each other.
if (prefix_order(prev.key, cur) != .eq) continue;
// Sorting puts equal keys next to each other.
if (!std.mem.eql(u8, prev.key, cur.key)) continue;
// A document's own entries were deduped at build time, so equal
// keys under one id are not a conflict — same rule as
// check_unique's exclude_id.
@@ -315,8 +320,11 @@ pub const Index = struct {
/// All ids whose key equals `key` (component-wise). For a partial key
/// (fewer components than the index has) this is a prefix search.
pub fn lookup_eq(self: *const Index, gpa: std.mem.Allocator, key: []const bson.Value, out: *std.ArrayListUnmanaged([]const u8)) !void {
const start = self.lower_bound_prefix(key);
const end = self.upper_bound_prefix(key);
var enc: std.ArrayListUnmanaged(u8) = .empty;
defer enc.deinit(gpa);
for (key) |v| try bson.encode_key(v, gpa, &enc);
const start = self.lower_bound_prefix(enc.items);
const end = self.upper_bound_prefix(enc.items);
var i = start;
while (i < end) : (i += 1) try out.append(gpa, self.entries.items[i].id);
}
@@ -338,27 +346,29 @@ pub const Index = struct {
// binary searches. Scanning the whole equality band and filtering
// made a range on the first component touch every entry in the
// index — O(n) for what is O(log n + result).
var ext: [max_index_keys]bson.Value = undefined;
@memcpy(ext[0..prefix.len], prefix);
var enc: std.ArrayListUnmanaged(u8) = .empty;
defer enc.deinit(gpa);
for (prefix) |v| try bson.encode_key(v, gpa, &enc);
const prefix_len = enc.items.len;
// Clamp into the equality band: lo/hi constrain only the component
// at prefix.len, so the bounds alone would reach past the entries
// that share the prefix.
var start = self.lower_bound_prefix(prefix);
var end = self.upper_bound_prefix(prefix);
var start = self.lower_bound_prefix(enc.items[0..prefix_len]);
var end = self.upper_bound_prefix(enc.items[0..prefix_len]);
if (lo) |l| {
ext[prefix.len] = l;
const key = ext[0 .. prefix.len + 1];
enc.items.len = prefix_len;
try bson.encode_key(l, gpa, &enc);
// Inclusive wants the first entry not less than lo; exclusive
// wants the first one strictly greater.
const bound = if (lo_incl) self.lower_bound_prefix(key) else self.upper_bound_prefix(key);
const bound = if (lo_incl) self.lower_bound_prefix(enc.items) else self.upper_bound_prefix(enc.items);
start = @max(start, bound);
}
if (hi) |h| {
ext[prefix.len] = h;
const key = ext[0 .. prefix.len + 1];
const bound = if (hi_incl) self.upper_bound_prefix(key) else self.lower_bound_prefix(key);
enc.items.len = prefix_len;
try bson.encode_key(h, gpa, &enc);
const bound = if (hi_incl) self.upper_bound_prefix(enc.items) else self.lower_bound_prefix(enc.items);
end = @min(end, bound);
}
if (start >= end) return;
@@ -368,13 +378,13 @@ pub const Index = struct {
/// First entry whose first `prefix.len` components are not less than
/// `prefix`.
fn lower_bound_prefix(self: *const Index, prefix: []const bson.Value) usize {
fn lower_bound_prefix(self: *const Index, prefix: []const u8) usize {
return std.sort.lowerBound(Entry, self.entries.items, prefix, prefix_order);
}
/// First entry whose first `prefix.len` components are greater than
/// `prefix`.
fn upper_bound_prefix(self: *const Index, prefix: []const bson.Value) usize {
fn upper_bound_prefix(self: *const Index, prefix: []const u8) usize {
return std.sort.upperBound(Entry, self.entries.items, prefix, prefix_order);
}
@@ -541,16 +551,16 @@ fn default_name(gpa: std.mem.Allocator, key_pairs: []const bson.Pair) ![]u8 {
// Comparison
// ---------------------------------------------------------------------------
/// Component-wise bson.compare, tie-broken by the id bytes. This is the
/// total order the entry array is kept in.
/// Encoded-key byte order, tie-broken by the id bytes. This is the total
/// order the entry array is kept in.
///
/// bson.encode_key guarantees this reproduces component-wise bson.compare
/// exactly, so ordering is a memcmp. Key direction is deliberately not
/// applied here: uniqueness and candidate generation are direction
/// independent, and sorting handles direction itself.
pub fn compare_entries(a: Entry, b: Entry) std.math.Order {
const n = @min(a.key.len, b.key.len);
for (a.key[0..n], b.key[0..n]) |av, bv| {
const o = bson.compare(av, bv);
const o = std.mem.order(u8, a.key, b.key);
if (o != .eq) return o;
}
const l = std.math.order(a.key.len, b.key.len);
if (l != .eq) return l;
return std.mem.order(u8, a.id, b.id);
}
@@ -558,15 +568,19 @@ fn entry_less(_: void, a: Entry, b: Entry) bool {
return compare_entries(a, b) == .lt;
}
/// Order of `prefix` against an entry key's leading components — `.eq` when
/// every prefix component matches (the key may be longer). The search-side
/// counterpart of compare_entries: same component-wise bson.compare, no
/// length or id tie-break, so a partial key matches a whole range.
fn prefix_order(prefix: []const bson.Value, e: Entry) std.math.Order {
for (prefix, e.key[0..prefix.len]) |p, k| {
const o = bson.compare(p, k);
/// Order of an encoded `prefix` against an entry key — `.eq` when the key
/// starts with it. The search-side counterpart of compare_entries: no id
/// tie-break, so a partial key matches a whole range.
///
/// Comparing raw bytes is sound because every column encoding is
/// self-delimiting, so a prefix of the encoded key is exactly the encoding
/// of its leading columns.
fn prefix_order(prefix: []const u8, e: Entry) std.math.Order {
const n = @min(prefix.len, e.key.len);
const o = std.mem.order(u8, prefix[0..n], e.key[0..n]);
if (o != .eq) return o;
}
// The key ran out first, so it sorts below the prefix.
if (prefix.len > e.key.len) return .gt;
return .eq;
}