diff --git a/README.md b/README.md index bdd2f98..5c7acce 100644 --- a/README.md +++ b/README.md @@ -123,9 +123,11 @@ the query planner to narrow scans. serialization-ambiguous (int32 1, int64 1, double 1.0 compare equal but hash differently — those fall back to a scan, as do string/symbol/code). -v1 limits: no index-accelerated sort, no hashed/text/geo/partial indexes, -and entry insert/removal is O(n) (a sorted array) — fine for a light -database, with a B-tree or id→entry map as the follow-up. A TTL sweep +v1 limits: no hashed/text/geo/partial indexes, and entry *insert* is O(n) +(a sorted array memmoves the tail) — fine for a light database, with a +B-tree as the follow-up. Removal is no longer a scan: entry generation is +a pure function of the document, so the entries to drop are regenerated +and found by binary search. A TTL sweep walks every entry of every TTL index and holds the write lock for the whole pass, so the interval is the tuning knob: expiry is never more precise than `--ttl-sweep-secs`, and a very large TTL index wants a @@ -154,21 +156,25 @@ the `tests/e2e/big.js` harness (12-core/32 GB Mac): 10-200x slower on every path (the matcher alone was 70 µs/doc in Debug vs 0.4 µs in ReleaseFast), which dwarfed every other difference in the MongoDB comparison below. -- **Compaction is O(n²) under the default 16 MB threshold.** A compaction - rewrites the whole log (one fsync per record), so bulk-loading 5 GB with - the default threshold degrades from ~310 MB/s to a crawl as the dataset - grows. Raise `--compact-threshold` for bulk loads — e.g. `2g` — and the - rate stays flat. The 5.37 GB run (40,960 × 128 KB docs, ObjectIds, - ReleaseFast) inserted in 36.5 s at ~310 MB/s between the two threshold - compactions, peaked at 5.25 GB RSS (~0.98x the data size at 128 KB - docs), and reopened the 5 GB log in 13.8 s. +- **Compaction no longer needs tuning for bulk loads.** It triggers on the + share of the log that is garbage rather than on bytes appended, so a pure + insert workload — which has no garbage — is never rewritten, and a + rewrite-heavy one is reclaimed once about a fifth of the log is dead, + keeping the file near 1.25x the live data. `--compact-threshold` is now + only a floor below which small logs are left alone. (It used to fire + every 16 MB regardless, rewriting the whole log each time: quadratic + total traffic, and the reason bulk loads needed a raised threshold.) - **`findOne({_id})` is O(1) only for ObjectId ids.** Integer, int64 and double ids compare equal but hash differently, so the docs-map fast path is skipped and every `_id` lookup becomes a full scan. Use the driver's - default ObjectIds (or a secondary index) on big collections. + default ObjectIds (or a secondary index) on big collections. The + order-preserving key encoding already removes the ambiguity that forces + this; lifting the restriction waits on an ordered `_id` index. - **Secondary-index entry insert is O(n)** (sorted array — see v1 limits - above), so creating an index over existing data or inserting with an - index in place is quadratic. Create indexes after the load. + above), so inserting into a collection that already has an index is + quadratic. Building an index over existing data is not: entries are + appended unsorted and ordered once. Still cheapest to create indexes + after the load. ## Performance vs MongoDB @@ -179,52 +185,87 @@ With the ReleaseFast default build (MongoDB 8.3.7 on the same Mac): | benchmark | mongo-lite | mongodb | winner | |---|---|---|---| -| insertOne (sequential) | 0.2 ms | 4.9 ms | **mongo-lite ×24** | -| bulk insert (insertMany) | 267 MB/s | 690 MB/s | mongodb ×2.6 | -| createIndex({k: 1}) | 0.66 s | 0.08 s | mongodb ×8 | -| countDocuments({}) | 2.5 ms | 11 ms | **mongo-lite ×4** | -| findOne({_id}) | 0.6 ms | 0.7 ms | mongo-lite | -| findOne indexed | 0.7 ms | 2.6 ms | **mongo-lite ×4** | -| range-scan count | 25 ms | 13 ms | mongodb ×2 | -| sort + limit(20) | 40 ms | 2 ms | mongodb ×20 | -| aggregate $group | 12 ms | 13 ms | mongo-lite | -| updateOne({_id}) | 0.18 ms | 0.21 ms | mongo-lite | -| updateMany (65 docs) | 20 ms | 6 ms | mongodb ×3 | -| deleteOne + insert | 0.9 ms | 5 ms | **mongo-lite ×6** | -| server RSS | 2.0 GB | 1.3 GB | mongodb (×0.65) | -| kill -9 → reopen | 3.8 s | 1.3 s | mongodb | -| db on disk | 1.0 GB | 89 MB | mongodb (compressed) | +| insertOne (sequential) | 0.19 ms | 5.0 ms | **mongo-lite ×26** | +| bulk insert (insertMany) | 853 MB/s | 690 MB/s | **mongo-lite ×1.2** | +| createIndex({k: 1}) | 62 ms | 78 ms | **mongo-lite** | +| countDocuments({}) | 1.5 ms | 11.5 ms | **mongo-lite ×8** | +| findOne({_id}) | 0.48 ms | 0.54 ms | mongo-lite | +| findOne indexed | 0.64 ms | 4.3 ms | **mongo-lite ×7** | +| range-scan count | 21 ms | 13 ms | mongodb ×1.7 | +| sort + limit(20), on `_id` | 4.3 ms | 2.0 ms | mongodb ×2 | +| sort + limit(20), indexed field | 1.0 ms | — | — | +| aggregate $group | 9.8 ms | 13.7 ms | **mongo-lite** | +| updateOne({_id}) | 0.16 ms | 0.19 ms | mongo-lite | +| updateMany (65 docs) | 5.5 ms | 6.1 ms | mongo-lite | +| deleteOne + insert | 0.62 ms | 4.9 ms | **mongo-lite ×8** | +| server RSS | 2.0 GB | 1.4 GB | mongodb (×0.7) | +| kill -9 → reopen | 0.8 s | 1.3 s | **mongo-lite** | +| db on disk | 1.0 GB | 93 MB | mongodb (compressed) | -The pattern: mongo-lite wins every *latency-bound* single-op (no network of -index hops, no journal latency, in-RAM) and loses the *throughput-bound* -bulk paths and the ops MongoDB accelerates with disk indexes and -compression. +The remaining losses are structural rather than incidental. Disk size is +the big one: payloads are stored raw, so the log is 11x MongoDB's +compressed files. RSS trails because every document carries its own arena. +The range-scan gap is not the matcher — it is walking 65,536 documents +that each live in a separate allocation, one pointer chase apiece. And +`sort` on `_id` still materializes candidates because nothing ordered +covers `_id` yet; the same sort on an indexed field streams straight out +of the index at 1.0 ms. -### Suggested improvements (highest impact first) +Reproduce the table with `bash tests/e2e/compare-run.sh 1g 16k`; the run +above is recorded in `tests/e2e/results/phase1.txt`. -1. **Index-accelerated sort** — the worst gap (×20): `sort+limit` sorts - every document. Stream candidates in index order (the planner already - has ordered range search) and stop at `limit`. Fixes the biggest read - regression. -2. **Batch index builds** — `createIndex` inserts entries one at a time - into a sorted array (O(n²) memmoves). Sort all entries once and append - in bulk (O(n log n)); a B-tree or id→entry map removes the O(n) entry - insert on the write path too. -3. **Compress the log** — the db is 11× MongoDB's on disk because payloads - are stored raw. Snappy per record (like the wire protocol's OP_COMPRESSED) - would shrink highly-compressible workloads massively. -4. **Faster reopen** — replay is a full re-parse of every record. A - periodic checkpoint record (or a parallel replay) would cut the 3× - restart gap. -5. **Trim the write path** — bulk insert (×2.6) is now bound by per-doc - parse/serialize/map-put, not fsync. A pooled per-connection arena for - owned docs and a bulk-insert fast path would close most of the gap; - updateMany's per-doc replace-serialize is the same story. -6. **Range-scan matching (×2)** — the matcher allocates a candidates list - per field per doc; a stack buffer for the common single-field case - removes it. +### What is left (highest impact first) -Two real bugs were found and fixed while benchmarking: +1. **Compress the log** — the largest remaining gap (×11). Payloads are + stored raw. A block-framed format with an LZ4 block codec would shrink + highly compressible workloads massively; note Zig 0.16 ships zstd + decompression only, and deflate would cap writes below the current + insert rate. +2. **A B-tree over the encoded keys** — entry insert still memmoves the + tail of a sorted array, so writing into a collection that already has an + index is quadratic. A flat, `u32`-indexed node array would also be + dumpable into a checkpoint, which is what makes a fast reopen possible. +3. **An ordered `_id` index** — `sort({_id: ...})` still materializes every + candidate, and integer `_id`s still scan. Both fall out of indexing the + encoded `_id`. It wants the tree first: an `_id` index updates on every + insert, and doing that against a sorted array is only cheap because + ObjectIds append at the end. +4. **Stop giving every document its own arena** — the source of both the + RSS gap and the range-scan gap. Storing canonical BSON bytes in a + per-collection slab and matching against them (parsing only the fields a + filter names) makes scans contiguous instead of a pointer chase. +5. **Decompose the global lock** — one reader/writer lock covers the whole + engine and is held across fsync, compaction and reply construction. + Per-collection locks plus cross-connection group commit are the path to + using more than one core on writes. + +Done so far, with the measurement that drove each: + +- **Record integrity hash CRC32 → XxHash3.** `std.hash.Crc32` is + table-driven and byte-at-a-time: 408 MB/s against XxHash3's 31 GB/s, or + 38 µs versus 0.5 µs on a 16 KB document — about two thirds of the entire + bulk-insert cost. Insert 260 → 700 MB/s. +- **Compaction triggers on garbage, not on bytes written**, and syncs once + per rewrite instead of once per document. Bulk load at the default + threshold 41.6 → 703 MB/s. +- **Index builds append then sort once** instead of inserting into a sorted + array. `createIndex` over 65,536 documents 649 → 44 ms. +- **Index entries hold encoded byte keys**, so comparing them is a memcmp + rather than a walk over values in unrelated arenas. +- **Entry removal is a binary search**, not a scan of the whole index. + `updateMany` 15.4 → 5.5 ms. +- **Top-k sort selection** and an allocation-free decorate pass, plus + **index-supplied ordering** when an index already holds candidates in the + requested order. `sort+limit(20)` 40 → 4.3 ms, or 1.0 ms on an indexed + field. +- **`limit` reaches the scan**, which used to materialize the whole + collection before slicing, and `countDocuments` is answered by counting + rather than by materializing and discarding every match. +- **Matching collects candidates on the stack**, resolves operators to an + enum once per filter field rather than by string per document, and reuses + one reply arena per connection. + +Several real bugs surfaced while benchmarking: - `plan_id` returned a pointer to a stack temporary (`&.{e}`) that dangled after the frame returned — Debug tolerated it, ReleaseFast read garbage, @@ -233,6 +274,11 @@ Two real bugs were found and fixed while benchmarking: - Multi-doc writes fsynced once per document; they now group-commit (one fsync per command, same crash guarantees — verified by the kill -9 crash suites). +- Compaction fsynced once per live document, because the log it wrote into + never had deferred syncing enabled — 65,536 fsyncs to rewrite a 1 GB + collection. +- `remove` never checked the compaction threshold, so a delete-heavy + workload grew the log without bound. ## Code style