dev/MultiforaDB
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

storage: byte documents in a per-collection slab (roadmap item 4)

Browse Source 
Documents live as canonical BSON bytes in a segmented per-collection slab
(fixed 8 MiB segments keep capacity slack under one segment); the docs map
holds flat offsets that stay valid across segment growth, and removed
documents leave garbage bytes until compaction rewrites. The per-document
ArenaAllocator and its second full Pair-tree copy are gone.

The matcher walks the stored bytes directly, skipping by length any field
the filter does not name (a new bson byte-walker: element_key, skip_value,
read_value with borrowed leaves, get_at, and a borrowed spine parse). The
byte matcher is differential-tested against the tree matcher on a corpus
and shares its operator logic. Stored documents are never materialized on
the scan path or in aggregate $match; $group reads group keys and sums
straight off the bytes. Sort, projection, findAndModify, updates and
index entry generation use a borrowed spine into the slab (or the byte
collector, which also replaced collect_values in build_entries). The
compaction threshold now counts uncompressed data volume, since a
compressed log would otherwise never trigger.

Measured (tests/e2e/results/phase5.txt): server RSS 1979 -> 539 MB (2.4x
smaller than MongoDB; phase1 baseline 2.0 GB), range-scan 22.5 -> ~12 ms
(parity, best run faster than MongoDB), proj 4.1 -> 3.4 ms, createIndex
parity. Verified: unit suite in all three modes with zero leaks, the
crash pair, e2e6, and the stress/spill programs.
This commit is contained in:
Alexey Shakhmatov
2026-08-02 22:15:07 +03:00
parent b4585106f1
commit 570900a6ef
12 changed files with 985 additions and 253 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
src/spill2.zig
View File

@@ -6,8 +6,11 @@ const std = @import("std");
const index = @import("index.zig");
const bson = @import("bson.zig");
fn doc_of(pairs: []const bson.Pair) bson.Document {
return .{ .arena = undefined, .pairs = pairs };
fn doc_of(gpa: std.mem.Allocator, pairs: []const bson.Pair) ![]u8 {
var out: std.ArrayListUnmanaged(u8) = .empty;
defer out.deinit(gpa);
try bson.write_doc(pairs, gpa, &out);
return out.toOwnedSlice(gpa);
}
pub fn main() !void {
@@ -42,8 +45,8 @@ pub fn main() !void {
try ids.append(gpa, id);
pairs[0] = .{ .key = "_id", .value = .{ .int32 = @intCast(i) } };
pairs[1] = .{ .key = "tag", .value = .{ .string = key } };
const d = doc_of(&pairs);
_ = try ix.add_doc(gpa, &d, id, false);
const d = try doc_of(gpa, &pairs);
_ = try ix.add_doc(gpa, d, id, false);
}
std.debug.print("count={d} leaves={d} depth={d} overflow={d}\n", .{ ix.count(), ix.leaf_count, ix.depth, ix.overflow.items.len });
if (ix.count() != N) return error.Bad;
@@ -70,8 +73,8 @@ pub fn main() !void {
for (order.items) |i| {
pairs[0] = .{ .key = "_id", .value = .{ .int32 = @intCast(i) } };
pairs[1] = .{ .key = "tag", .value = .{ .string = facts.items[i].key } };
const d = doc_of(&pairs);
ix.remove_doc(gpa, &d, ids.items[i]);
const d = try doc_of(gpa, &pairs);
ix.remove_doc(gpa, d, ids.items[i]);
removed += 1;
if (ix.count() != N - removed) {
std.debug.print("count mismatch at {d}: {d} != {d}\n", .{ i, ix.count(), N - removed });