Aleksey Shakhmatov
ecd28d9b26
The single engine-wide reader/writer lock is replaced by a lock hierarchy, so writes to different collections no longer serialize on one mutex: - Collections are heap-allocated, so their addresses are stable while a command holds a collection lock (the maps only store pointers). - A catalog rwlock guards the database/collection maps: shared for every command (so a concurrent DDL cannot mutate the maps underneath it), exclusive for create/drop/dropDatabase. Each collection has its own rwlock; the ordering is always catalog -> collection -> log lock, never two collection locks at once (TTL sweep and compaction take collections one at a time). - Command dispatch acquires the catalog + target collection locks for the handler's duration, resolving the collection (creating it for writes) under the catalog lock; create/drop upgrade to the exclusive catalog lock. - Appends never fsync. Each write command's epilogue releases the collection lock, then commits once (seal + fsync) with a leader/follower group commit: the leader waits for writers mid-append (a pending counter) so its seal covers them, and followers whose records the seal covered skip their own fsync. Every acknowledged write is fsynced before its reply (crash pair verified); an unacknowledged write may vanish and a reader may observe a write before its fsync — ordinary w:1 j:true semantics instead of 'the log describes >= memory'. - Compaction snapshots collections without the log lock (so a concurrent writer holding one can always finish its append) and retries when a writer appended mid-snapshot (detected via the record seq), then swaps under the log lock — no deadlock. The compaction trigger moved to the command epilogue and the TTL monitor. - Engine.dup_index moved to the collection (per-command error paths). Also lands two B-tree edge-case fixes driven by tests that were in flight: a churned leaf full of dead bytes no longer splits with an empty right half (the leaf is repacked before splitting, and an emptied node's page is fully free again), and a slot-count split with all large records on one side shifts records between the halves until the new record fits. Plus a randomised fuzz test over key sizes (src/fuzz_split.zig) and the two regression tests. Measured (tests/e2e/results/phase6.txt): no regression on the single-connection benchmark; concurrent durable-insert throughput ~5.1k -> 12.5k docs/s from 1 -> 8 clients, ~14.8k at 32. Verified: unit suite in all three modes, all e2e suites, the kill -9 crash pair.
50 lines
3.6 KiB
Plaintext
50 lines
3.6 KiB
Plaintext
# Phase 6 gate — mongo-lite vs MongoDB 8.3.7, 1g dataset / ~16k docs
|
||
# Ratio < 1.0 = mongo-lite faster. Reproduce: bash tests/e2e/compare-run.sh 1g 16k
|
||
# This run includes all five roadmap items (B+tree, _id index, compressed
|
||
# log, byte storage, decomposed locks). Compare: tests/e2e/results/phase1.txt.
|
||
|
||
benchmark mongo-lite mongodb ratio
|
||
insertOne (sequential) ×200 0.20 ms 4.7 ms 0.0x
|
||
bulk insert throughput 739.0 MB/s 694.5 MB/s 1.1x
|
||
docs loaded 65,536 65,536 1.0x
|
||
createIndex({k: 1}) 64.7 ms 82.1 ms 0.8x
|
||
countDocuments({}) 3.4 ms 10.9 ms 0.3x
|
||
findOne({_id: <ObjectId>}) 0.83 ms 0.72 ms 1.2x
|
||
findOne({k: 500}) (indexed) 0.62 ms 1.6 ms 0.4x
|
||
find({p: {$gte,$lt}}).count() (scan) 12.4 ms 12.0 ms 1.0x
|
||
find({}).sort({_id:-1}).limit(20) 2.3 ms 2.1 ms 1.1x
|
||
find({}, {proj}).limit(1000) 3.7 ms 4.3 ms 0.9x
|
||
aggregate $group by k 10.2 ms 12.8 ms 0.8x
|
||
updateOne({_id}) ×50 0.17 ms 0.20 ms 0.8x
|
||
updateMany({k: 7}, {$inc}) 1.9 ms 6.3 ms 0.3x
|
||
deleteOne({_id}) + insertOne 0.58 ms 5.1 ms 0.1x
|
||
node client RSS 152 MB 157 MB 1.0x
|
||
server RSS 547 MB 1274 MB
|
||
kill -9 reopen 0.8s 1.3s
|
||
db on disk 97MB 88MB
|
||
|
||
# Item 5 (decomposed locks) deltas vs phase5: none on this single-connection
|
||
# benchmark (all rows within run noise). The structure is the deliverable:
|
||
# - collections are heap-allocated (stable pointers), the docs/slab/index
|
||
# maps are guarded by a catalog rwlock (shared for commands, exclusive
|
||
# for create/drop) plus one rwlock per collection (catalog -> collection
|
||
# ordering, one collection at a time for TTL sweep and compaction).
|
||
# - appends never fsync; each write command's epilogue commits once
|
||
# (seal + fsync) with leader/follower group commit: the leader waits
|
||
# for writers mid-append, seals, and syncs once, and followers whose
|
||
# records the seal covered skip their own fsync.
|
||
# - compaction snapshots collections one at a time without the log lock
|
||
# and retries if a writer appended during the snapshot (no deadlock
|
||
# against a writer holding a collection lock), then swaps under the log
|
||
# lock.
|
||
# - durability semantics: acknowledged writes are fsynced before their
|
||
# reply (crash pair verified); an unacknowledged write may vanish, and
|
||
# a reader can observe a write before its fsync completes — ordinary
|
||
# w:1 j:true semantics, no longer "the log describes >= memory".
|
||
#
|
||
# Concurrent-write throughput (sequential insertOne per client, durable):
|
||
# 1 client ~5.1k docs/s | 8 clients ~12.5k docs/s | 32 clients ~14.8k
|
||
# docs/s. The fsync per commit still dominates sequential-per-client
|
||
# workloads; the group commit coalesces when appends overlap (different
|
||
# collections on different connections).
|