index: the node arena and overflow slab live in the data file
The last structures move onto the pager, so the whole engine's storage is now
one mapped file plus the WAL.
Node ids are deliberately *not* page numbers. PLAN amendment A1 explains why:
`Node.parent`, `next`, `prev` and an internal slot's `extra` are back-pointers
by id, so copy-on-write moving a page would force every node referring to it to
move as well -- COWing one leaf cascades through the leaf level, one internal
node through its whole subtree. An in-RAM id->page table makes the table slot
the single owner of a page number, so COW has exactly one pointer to fix. It
costs one dependent load per node access and 4 bytes per node, about 5.6 MB at
100M documents, against the 64-100 bytes *per document* this milestone removes.
The overflow slab becomes extents too, so `Slot.off` for a spilled record is an
absolute file offset -- the same change documents went through.
--
Two bugs, both found by measuring rather than by reading, and both worth
recording because the second one would have been invisible until the churn gate.
The reservation was a tail mark, and it cannot be: an upsert reserves tree pages
for every index *and* slab room for the document, all before one log append. The
second reserver overwrote the first one's promise and the first one's allocation
then asserted. Caught on a 512 MB load by the tripwire added in the
`reserve_for` commit, which is the entire reason that assert exists. It is a
count now, and the multi-consumer ordering is pinned by a test.
And a reservation was never released. It is scoped to one write -- taken before
the log append so the publish cannot fail -- but a tree reservation covers the
worst case of several splits while a typical insert causes none, so the promise
accumulated by a handful of pages per write and dragged the file up with it. The
data file was **1.89 GB for 512 MB of documents**; releasing the unclaimed
promise at the end of each write brings it to 551 MB, or 1.08x, which is the
extent slack and the node pages.
--
Measured on one harness, 512 MB / 16 KB docs, against the in-RAM engine this
replaces:
bulk insert throughput 742.6 MB/s -> 736.4 MB/s
insertOne (sequential) 0.20 ms -> 0.22 ms
createIndex({k: 1}) 26.8 ms -> 16.5 ms
countDocuments({}) 2.1 ms -> 1.1 ms
findOne({k: 500}) indexed 0.75 ms -> 0.56 ms
find({p: range}).count() 6.6 ms -> 4.6 ms
aggregate $group by k 5.8 ms -> 3.8 ms
updateMany({k: 7}, {$inc}) 1.2 ms -> 1.0 ms
Reads gain from one contiguous mapping; the two write rows are within noise of
flat. RSS is still unchanged and still cannot improve, for the reason given in
the previous commit: every open replays the whole log and rebuilds everything.
The dev harnesses each open their own data file now. `zig build fuzz` caught all
four of them, again.
This commit is contained in:
12
src/db.zig
12
src/db.zig
@@ -86,7 +86,7 @@ pub const Collection = struct {
|
||||
// where serialize_value is not, so int32 1 / int64 1 / double 1.0
|
||||
// collide as they do in MongoDB -- see the migration note in
|
||||
// apply_record.
|
||||
self.id_index = try index.Index.init(gpa, "_id_", &keys, true, false, null);
|
||||
self.id_index = try index.Index.init(gpa, pager, "_id_", &keys, true, false, null);
|
||||
return self;
|
||||
}
|
||||
|
||||
@@ -687,6 +687,9 @@ pub const Engine = struct {
|
||||
b.ix.insert_entries(&b.built, off);
|
||||
}
|
||||
stored = true;
|
||||
// The write is published; anything the reservations above did not claim
|
||||
// is dead. Leaving it promised would grow the file on every write.
|
||||
self.pager.release_reservation();
|
||||
self.note_compact();
|
||||
}
|
||||
|
||||
@@ -771,7 +774,7 @@ pub const Engine = struct {
|
||||
spec_doc: *const bson.Document,
|
||||
) !*index.Index {
|
||||
const coll = try self.get_or_create_collection(db_name, coll_name);
|
||||
const parsed = try index.parse_spec(self.gpa, spec_doc);
|
||||
const parsed = try index.parse_spec(self.gpa, self.pager, spec_doc);
|
||||
// Boxed before anything is built into it, so publishing is a pointer
|
||||
// append rather than a struct copy. An Index will own a mapping once
|
||||
// the arena is file-backed, and copying one then would duplicate that
|
||||
@@ -806,6 +809,7 @@ pub const Engine = struct {
|
||||
try ix.append_doc_entries(self.gpa, coll.doc_bytes(entry.value_ptr.*), entry.value_ptr.*);
|
||||
}
|
||||
_ = try ix.finish_bulk(self.gpa, true);
|
||||
self.pager.release_reservation();
|
||||
|
||||
// Reserve the collection slot, then persist and publish.
|
||||
try coll.indexes.ensureUnusedCapacity(self.gpa, 1);
|
||||
@@ -1237,6 +1241,7 @@ pub const Engine = struct {
|
||||
};
|
||||
}
|
||||
// Tolerated, not enforced: the database must always open.
|
||||
defer self.pager.release_reservation();
|
||||
if (try ix.finish_bulk(self.gpa, false)) {
|
||||
std.debug.print(
|
||||
"multiforadb: WARNING: unique index '{s}' has duplicate keys in existing " ++
|
||||
@@ -1255,7 +1260,7 @@ pub const Engine = struct {
|
||||
coll: *Collection,
|
||||
spec_doc: *const bson.Document,
|
||||
) !void {
|
||||
const parsed = try index.parse_spec(self.gpa, spec_doc);
|
||||
const parsed = try index.parse_spec(self.gpa, self.pager, spec_doc);
|
||||
const ix = self.gpa.create(index.Index) catch |err| {
|
||||
var dead = parsed;
|
||||
dead.deinit(self.gpa);
|
||||
@@ -1337,6 +1342,7 @@ fn apply_record(ctx: *anyopaque, record: storage.Record, doc: *bson.Document) an
|
||||
defer self.gpa.free(doc_bytes);
|
||||
try coll.slab_reserve(self.gpa, doc_bytes.len);
|
||||
const off = coll.slab_append(doc_bytes);
|
||||
self.pager.release_reservation();
|
||||
try coll.docs.put(self.gpa, id_key, off);
|
||||
self.live_docs += 1;
|
||||
key_owned = true;
|
||||
|
||||
Reference in New Issue
Block a user