dev/MultiforaDB
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

query/commands/wire: trim the scan and request paths

Browse Source 
Matching allocated an ArrayList per filter field per candidate document,
on the process-wide allocator, to hold what is almost always a single
value. Candidates now collect into a stack buffer that spills to the heap
only for arrays: measured 15.7 -> 12.0ms on a 65,536-document range scan.

The OOM-propagation test moves with it. Its point is that a failed
collection must surface as an error rather than an empty candidate list,
which would make $ne and $exists:false report a match -- a wrong answer
rather than a failed one. That invariant still holds on the spill path, so
the test now uses an array long enough to reach the allocator, and a new
test pins the flip side: the common single-value match now completes
correctly even when the allocator always fails, because it never calls it.

Query operators were dispatched by a chain of up to fourteen mem.eql per
value per document, with $gt/$gte/$lt/$lte re-comparing the operator name
inside the loop over candidate values. Names resolve to an enum once per
filter field. Command dispatch likewise walked a 30-entry table comparing
strings; it is a comptime StaticStringMap now.

Each request built a fresh reply arena and handed its pages straight back.
One reply per connection, reset between requests, keeps them.

countDocuments() arrives as [{$match: F}?, {$group: {_id: <literal>,
n: {$sum: 1}}}], which the general path answered by materializing every
matching document and discarding them all. It is now recognized and
answered from a counting scan: countDocuments({}) 2.3 -> 1.5ms.

The detector is deliberately conservative -- grouping by "$field", summing
a field, an unmodelled accumulator or any extra stage all fall through to
the general path, since those need the documents themselves. A unit test
pins each accept and reject, and the whole count path was checked against
the general one through the real driver, including the shapes that must
not take it.

The filtered range-scan row does not move: it is bound by walking 65,536
documents that each live in their own arena, not by the matcher. That is
Phase 4 work.

Verified: 77 unit tests under ReleaseFast and ReleaseSafe, e2e
29/16/17/3/2, the crash pair, e2e6 72/72.
This commit is contained in:
Alexey Shakhmatov
2026-08-02 19:13:29 +03:00
parent 552b916833
commit 75e412a4af
5 changed files with 327 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files

49
tests/e2e/results/phase1.txt
View File

@@ -1,34 +1,39 @@
# Phase 1 gate — mongo-lite vs MongoDB 8.3.7, 1g dataset / ~16k docs
# ratio < 1.0 = mongo-lite faster. Reproduce: bash tests/e2e/compare-run.sh 1g 16k
# commit: 9eecb50
# Run-to-run noise is roughly +/-15% on the scan rows; mongod's own numbers
# moved that much across the runs below.
benchmark mongo-lite mongodb ratio
insertOne (sequential) x200 0.19 ms 5.1 ms 0.0x
bulk insert throughput 755.6 MB/s 688.7 MB/s 1.1x
insertOne (sequential) x200 0.19 ms 5.0 ms 0.0x
bulk insert throughput 852.9 MB/s 689.6 MB/s 1.2x
docs loaded 65,536 65,536 1.0x
createIndex({k: 1}) 60.2 ms 80.6 ms 0.7x
countDocuments({}) 2.1 ms 13.8 ms 0.2x
findOne({_id: <ObjectId>}) 0.53 ms 0.84 ms 0.6x
findOne({k: 500}) (indexed) 0.55 ms 1.7 ms 0.3x
find({p: {$gte,$lt}}).count() (scan) 20.5 ms 13.1 ms 1.6x
find({}).sort({_id:-1}).limit(20) 6.7 ms 2.5 ms 2.7x
find({}, {proj}).limit(1000) 3.5 ms 4.4 ms 0.8x
aggregate $group by k 10.3 ms 12.9 ms 0.8x
updateOne({_id}) x50 0.14 ms 0.19 ms 0.7x
updateMany({k: 7}, {$inc}) 16.2 ms 6.5 ms 2.5x
deleteOne({_id}) + insertOne 0.61 ms 5.0 ms 0.1x
node client RSS 152 MB 156 MB 1.0x
server RSS 1975 MB 1379 MB
createIndex({k: 1}) 62.4 ms 78.4 ms 0.8x
countDocuments({}) 1.5 ms 11.5 ms 0.1x
findOne({_id: <ObjectId>}) 0.48 ms 0.54 ms 0.9x
findOne({k: 500}) (indexed) 0.64 ms 4.3 ms 0.1x
find({p: {$gte,$lt}}).count() (scan) 21.4 ms 12.7 ms 1.7x
find({}).sort({_id:-1}).limit(20) 7.1 ms 2.0 ms 3.5x
find({}, {proj}).limit(1000) 3.7 ms 4.3 ms 0.9x
aggregate $group by k 9.8 ms 13.7 ms 0.7x
updateOne({_id}) x50 0.16 ms 0.19 ms 0.8x
updateMany({k: 7}, {$inc}) 17.3 ms 6.1 ms 2.8x
deleteOne({_id}) + insertOne 0.62 ms 4.9 ms 0.1x
node client RSS 160 MB 164 MB 1.0x
server RSS 1974 MB 1386 MB
kill -9 reopen 0.8s 1.3s
db on disk 1025MB 92MB
db on disk 1025MB 93MB
# Baseline before Phase 1, for reference:
# bulk insert 267 MB/s | createIndex 0.66s | sort+limit 40ms
# Baseline before Phase 1:
# bulk insert 267 MB/s | createIndex 0.66s | sort+limit 40ms | countDocuments 2.5ms
# range-scan 25ms | updateMany 20ms | reopen 3.8s | disk 1.0GB | RSS 2.0GB
#
# Remaining gaps and where they are addressed:
# db on disk 11x -> Phase 3 (block-compressed log)
# sort+limit 2.7x -> Phase 2 (index-ordered scan, _id as an ordered index)
# updateMany 2.5x -> Phase 2 (B+tree: remove_id is a linear scan per index)
# range-scan 1.6x -> 1.9 (per-field-per-doc allocation in the matcher)
# sort+limit 3.5x -> Phase 2 (index-ordered scan, _id as an ordered index)
# updateMany 2.8x -> Phase 2 (B+tree; remove_id is a linear scan per index)
# range-scan 1.7x -> Phase 4, not the matcher. The stack-buffer change
# measured 15.7 -> 12.0ms in isolation, but this row is
# bound by walking 65,536 documents that each live in
# their own arena: hash-map iteration plus a pointer
# chase per document. Contiguous byte storage is the fix.
# server RSS 1.4x -> Phase 4 (per-document arena -> byte storage)