storage: byte documents in a per-collection slab (roadmap item 4)
Documents live as canonical BSON bytes in a segmented per-collection slab (fixed 8 MiB segments keep capacity slack under one segment); the docs map holds flat offsets that stay valid across segment growth, and removed documents leave garbage bytes until compaction rewrites. The per-document ArenaAllocator and its second full Pair-tree copy are gone. The matcher walks the stored bytes directly, skipping by length any field the filter does not name (a new bson byte-walker: element_key, skip_value, read_value with borrowed leaves, get_at, and a borrowed spine parse). The byte matcher is differential-tested against the tree matcher on a corpus and shares its operator logic. Stored documents are never materialized on the scan path or in aggregate $match; $group reads group keys and sums straight off the bytes. Sort, projection, findAndModify, updates and index entry generation use a borrowed spine into the slab (or the byte collector, which also replaced collect_values in build_entries). The compaction threshold now counts uncompressed data volume, since a compressed log would otherwise never trigger. Measured (tests/e2e/results/phase5.txt): server RSS 1979 -> 539 MB (2.4x smaller than MongoDB; phase1 baseline 2.0 GB), range-scan 22.5 -> ~12 ms (parity, best run faster than MongoDB), proj 4.1 -> 3.4 ms, createIndex parity. Verified: unit suite in all three modes with zero leaks, the crash pair, e2e6, and the stress/spill programs.
This commit is contained in:
@@ -102,6 +102,10 @@ pub const Log = struct {
|
||||
path: []const u8,
|
||||
end_pos: u64,
|
||||
log_bytes: u64, // bytes written since the log was last rewritten
|
||||
/// Uncompressed record bytes appended since the log was last rewritten —
|
||||
/// the data volume, which the compaction threshold is really about (the
|
||||
/// on-disk size shrinks with compression and would under-trigger).
|
||||
data_bytes: u64 = 0,
|
||||
codec: u8,
|
||||
// Reused record-framing buffer. Appends are single-writer (the engine's
|
||||
// exclusive lock), so one buffer avoids a realloc cycle per record.
|
||||
@@ -224,7 +228,6 @@ pub const Log = struct {
|
||||
// runs on input already proven intact.
|
||||
const file_len = self.file.length(self.io) catch return error.InvalidLog;
|
||||
if (pos + total >= file_len) return;
|
||||
std.debug.print("mongo-lite: corrupt block hash at {d}\n", .{pos});
|
||||
return error.InvalidLog;
|
||||
}
|
||||
|
||||
@@ -331,6 +334,7 @@ pub const Log = struct {
|
||||
std.mem.writeInt(u32, buf.items[0..4], total, .little);
|
||||
std.mem.writeInt(u64, buf.items[4..12], record_hash(buf.items[12..]), .little);
|
||||
|
||||
self.data_bytes += buf.items.len;
|
||||
// Seal the current block when the next record would push it past the
|
||||
// target; a single oversized record keeps its own block.
|
||||
if (self.block.items.len > 0 and self.block.items.len + buf.items.len > block_target) {
|
||||
|
||||
Reference in New Issue
Block a user