results: the M0 gates, measured

PLAN D7's six items, with the numbers and the command that reproduces each in
tests/e2e/results/m0-gates.txt. Unit tests green in both optimize modes, the
whole e2e matrix green, the spec scorecard byte-identical at 131/161/195, and
the large smoke run at the scale D7.3 asked for:

  21.47 GB collection (1,310,720 x 16 KiB)
  data file                 21.75 GB      (+1.3% over the documents)
  log after the load        2.5 MB        (checkpoints reclaim it)
  kill -9 then reopen       0.5 s         (0.5 s at 4 GB too -- flat)
  RSS after reopen          237 MB        (1.1% of the data)
  count after restart       1,310,720     last document byte-intact
  acked writes after kill   200/200

That is the milestone's claim, measured: an open costs the working set rather
than the size of the database. Before M0 the same measurement was 523 MB
resident for a 512 MB database, because recovering each document's `_id` meant
reading every document at open.

Two gates need reading rather than a tick, and m0-gates.txt says so where a
reader would otherwise take a tick for granted.

The churn gate settles at 1.65x live data (delete-heavy) to 2.47x
(update-heavy), flat, above the ~1.3x amendment A2 hoped for. Rebuild-only
reclamation cannot reach that: it needs a whole second copy of the live data
before the first can be freed. The gate existed to decide whether doc-level
free lists are needed after M0, and that is the answer.

Benchmark parity holds for every read and latency row inside the run-to-run
spread, and bulk insert regresses 24% (732 -> 555 MB/s), reproducibly across
three runs. Risk 1 as written: document bytes now reach the disk uncompressed
on top of the LZ4 log. createIndex improves 62% from the same change.

Three measurement bugs fixed while running the gates, because each would have
put a false number in the README:

  - `compare-run.sh` measured "db on disk" as `du` of the log alone against
    `du` of mongod's whole dbpath. It reported 20 MB for a 1 GB collection --
    the documents had moved to <db>.data. Honest figure, measured: 914 MB of
    allocated blocks against mongod's compressed 85 MB.
  - `big.js` counted "compaction events" as "the log shrank", which is a
    *checkpoint* now. It claimed 12 compaction rewrites during a pure insert
    load, which has no garbage to compact.
  - `big.js` labelled peak RSS "in-memory engine: docs live in RAM" and its
    summary said the collection was held "fully in RAM". Both were true of the
    engine this milestone replaced.

README: the storage section described an all-in-RAM engine; the comparison
table mixed one old run's body with three new rows; and `findOne({_id})` was
documented as a full scan for integer ids, which the ordered `_id_` index made
false (2 ms against 55 s for a scan of the same 21.5 GB collection). The table
is now best-of-three for both servers, with the measured variance stated, since
two runs of the same binary moved the sub-10 ms rows by 27-51%.

README.md

@@ -10,7 +10,10 @@ driver, PyMongo — connect over TCP and just work.
 The direction from this MVP — a full-fledged embedded, tens-of-GB,
 maximally MongoDB-compatible database — its decision record, milestones
 and gates live in [PLAN.md](PLAN.md). Milestone 0 (mmap + WAL storage
-foundation) is next.
+foundation) has landed; its measured gate results are in
+[`tests/e2e/results/m0-gates.txt`](tests/e2e/results/m0-gates.txt).
+Milestone 1 (cursors, and the doc-level free list the churn gate showed is
+needed) is next.
 
 ## Quick start
 
@@ -155,11 +158,42 @@ longer one.
 
 ## Working with large collections
 
-Everything lives in RAM (db → collection → _id → document maps) and every
-write command is logged with `fsync` before it is acknowledged (one sync per
-command via group commit — a 500-doc `insertMany` syncs once, not 500
-times), so multi-GB collections work, with cost/behavior notes measured by
-the `tests/e2e/big.js` harness (12-core/32 GB Mac):
+Documents, B+tree pages and overflow records live in an mmap'd data file
+(`<db>.data`), with the append-only log as the write-ahead log in front of it.
+Every write command is logged with `fsync` before it is acknowledged (one sync
+per command via group commit — a 500-doc `insertMany` syncs once, not 500
+times); a periodic checkpoint publishes the data file and truncates the log.
+So resident memory is the working set rather than the size of the database, and
+an open does not replay everything ever written. Measured by the
+`tests/e2e/big.js` harness on a 12-core/32 GB Mac:
+
+| 21.5 GB collection (1.3M × 16 KiB) | |
+|---|---|
+| data file | 21.75 GB (+1.3% over the documents) |
+| log after the load | 2.5 MB — checkpoints reclaim it |
+| `kill -9` then reopen | 0.5 s (also 0.5 s at 4 GB) |
+| resident after reopen | 237 MB — 1.1% of the data |
+| documents after restart | all 1,310,720, last one byte-intact |
+| acked writes surviving `kill -9` | 200/200 |
+
+Notes on the cost side, from the same run:
+
+- **A bulk load still touches everything it writes.** Peak resident during the
+  20 GB load was 18.5 GB: writing 21 GB of pages dirties 21 GB of pages, and
+  the kernel keeps them until it wants the memory back. The mmap win is in
+  reopen and steady-state reads, not in bulk ingest.
+- **Bulk insert costs about a quarter of its old throughput** (732 → 555 MB/s
+  at 1 GB), because document bytes now reach the disk uncompressed in the data
+  file on top of the compressed log. This was the anticipated trade for the
+  rows above; see `m0-gates.txt` for the untried mitigations.
+- **A cold full scan reads the whole collection from disk** — ~55 s for 21 GB,
+  about 390 MB/s. Index the fields you filter on; `countDocuments({})` with no
+  filter is a full scan by definition.
+- **Churn is bounded but not tight.** Under sustained rewriting the data file
+  settles at 1.65× (delete-heavy) to 2.47× (update-heavy) the live data and
+  stays there. Reclamation is by whole-collection rebuild, which needs a second
+  copy of the live data before it can free the first; a doc-level free list is
+  the M1 fix.
 
 - **Build in ReleaseFast** — `zig build` defaults to it. A Debug server is
   10-200x slower on every path (the matcher alone was 70 µs/doc in Debug
@@ -173,12 +207,16 @@ the `tests/e2e/big.js` harness (12-core/32 GB Mac):
   only a floor below which small logs are left alone. (It used to fire
   every 16 MB regardless, rewriting the whole log each time: quadratic
   total traffic, and the reason bulk loads needed a raised threshold.)
-- **`findOne({_id})` is O(1) only for ObjectId ids.** Integer, int64 and
-  double ids compare equal but hash differently, so the docs-map fast path
-  is skipped and every `_id` lookup becomes a full scan. Use the driver's
-  default ObjectIds (or a secondary index) on big collections. The
-  order-preserving key encoding already removes the ambiguity that forces
-  this; lifting the restriction waits on an ordered `_id` index.
+- **`findOne({_id})` is an index descent for every `_id` type.** It used to
+  be O(1) for ObjectIds and a *full scan* for integer, int64 and double ids,
+  which compare equal but hashed differently. The hash map is gone: `_id_` is
+  an ordered B+tree over the canonical key encoding, so all of those are one
+  descent. Measured on the 21.5 GB collection with integer ids:
+  `findOne({_id})` 2 ms, against 55 s for a scan of the same collection.
+  One consequence to know about: because the encoding is canonical, `1`
+  (int32), `1` (int64) and `1.0` (double) are now the *same* `_id` — which
+  matches MongoDB, and which a database written by an older build will warn
+  loudly about on first open if it holds two such documents.
 - **Secondary-index entry insert is O(n)** (sorted array — see v1 limits
   above), so inserting into a collection that already has an index is
   quadratic. Building an index over existing data is not: entries are
@@ -194,37 +232,65 @@ With the ReleaseFast default build (MongoDB 8.3.7 on the same Mac):
 
 | benchmark | MultiforaDB | mongodb | winner |
 |---|---|---|---|
-| insertOne (sequential) | 0.20 ms | 4.7 ms | **MultiforaDB ×24** |
-| bulk insert (insertMany) | 752 MB/s | 744 MB/s | MultiforaDB |
-| createIndex({k: 1}) | 67 ms | 76 ms | **MultiforaDB** |
-| countDocuments({}) | 2.6 ms | 11.2 ms | **MultiforaDB ×4** |
-| findOne({_id}) | 0.45 ms | 0.65 ms | **MultiforaDB** |
-| findOne indexed | 0.54 ms | 4.6 ms | **MultiforaDB ×8** |
-| range-scan count | 13.7 ms | 12.6 ms | mongodb ×1.1 |
-| sort + limit(20), on `_id` | 2.3 ms | 2.0 ms | mongodb ×1.1 |
-| sort + limit(20), indexed field | 1.0 ms | — | — |
-| aggregate $group | 8.1 ms | 12.3 ms | **MultiforaDB** |
-| updateOne({_id}) | 0.15 ms | 0.19 ms | **MultiforaDB** |
-| updateMany (65 docs) | 1.7 ms | 6.1 ms | **MultiforaDB ×3.6** |
-| deleteOne + insert | 0.50 ms | 4.9 ms | **MultiforaDB ×10** |
-| server RSS | 539 MB | 1.3 GB | **MultiforaDB ×2.4** |
-| kill -9 → reopen | 0.8 s | 1.3 s | **MultiforaDB** |
-| db on disk | 97 MB | 91 MB | mongodb |
+| insertOne (sequential) ×200 | 0.20 ms | 4.4 ms | **MultiforaDB ×22** |
+| bulk insert (insertMany) | 555 MB/s | 739 MB/s | mongodb ×1.3 |
+| createIndex({k: 1}) | 27.9 ms | 70 ms | **MultiforaDB ×3** |
+| countDocuments({}) | 2.6 ms | 11.3 ms | **MultiforaDB ×4** |
+| findOne({_id}) | 0.60 ms | 0.61 ms | parity |
+| findOne indexed | 0.54 ms | 1.1 ms | **MultiforaDB ×2** |
+| range-scan count | 11.2 ms | 12.5 ms | **MultiforaDB** |
+| sort + limit(20) on `_id` | 1.5 ms | 2.0 ms | **MultiforaDB** |
+| projection + limit(1000) | 3.6 ms | 4.3 ms | **MultiforaDB** |
+| aggregate $group | 7.4 ms | 13.4 ms | **MultiforaDB ×2** |
+| updateOne({_id}) ×50 | 0.15 ms | 0.21 ms | **MultiforaDB** |
+| updateMany (65 docs) | 1.9 ms | 5.9 ms | **MultiforaDB ×3** |
+| deleteOne + insert | 0.63 ms | 4.9 ms | **MultiforaDB ×8** |
+| concurrent durable writes, 32 clients | 32,817/s | 3,765/s | **MultiforaDB ×9** |
+| server RSS after the load | 1.06 GB | 1.18 GB | MultiforaDB |
+| kill -9 → reopen | 0.3 s | 1.3 s | **MultiforaDB ×4** |
+| db on disk | 914 MB | 85 MB | **mongodb ×11** |
 
-The engine now holds every document as canonical BSON bytes in a
-segmented per-collection slab (no per-document arena, no second Pair-tree
-copy), which is why RSS is a quarter of MongoDB's and the range scan —
-matching against the bytes directly, skipping fields by length — runs at
-parity. The log is LZ4-compressed in 256 KiB blocks, so the on-disk size
-matches MongoDB's compressed files. Bulk insert is compress-bound (the
-LZ4 codec runs at ~1.7 GB/s; deflate would cap writes below the insert
-rate, which is why the roadmap chose LZ4).
+Each cell is the best of three runs of the same suite, for both servers. That
+is not fussiness: two consecutive runs of the *same* binary moved the sub-10 ms
+rows by 27–51% on this machine, so a single run's ratios say more about the
+minute they were taken in than about either database. Treat differences under
+about 1.5× as noise.
 
-Reproduce the table with `bash tests/e2e/compare-run.sh 1g 16k`; the
-pre-tree baseline is recorded in `tests/e2e/results/phase1.txt`, and the
-runs with the B+tree, ordered `_id` index, compressed log and byte
-storage (roadmap items 1–4) in `tests/e2e/results/phase2.txt` through
-`phase5.txt`.
+Two rows in that table changed direction with the mmap foundation and are
+worth being explicit about.
+
+**`db on disk`** was 97 MB against mongod's 91 MB when the log was the only
+copy of the data and LZ4 compressed it. The data file does not compress
+documents: 65,536 × 16 KiB documents now occupy 914 MB of allocated blocks
+(`du`; `ls` shows 1.09 GB, the difference being the sparse tail the file is
+grown into) against mongod's compressed 85 MB. Per-page or per-extent
+compression is the fix, and it is not in M0. If you are comparing against a
+report from before this was written, note that the row used to measure the log
+file alone — which said 20 MB for a 1 GB collection, because the documents had
+moved to `<db>.data`. Fixed in `compare-run.sh`.
+
+**`server RSS`** is measured right after writing the whole dataset, so it
+reflects a bulk load having dirtied every page it wrote, not steady state. The
+number that speaks to the architecture is resident memory *after a reopen*:
+237 MB for a 21.5 GB collection (see the section above). Before M0 the same
+measurement was 523 MB for a 512 MB collection, because recovering each
+document's `_id` read every document at open.
+
+The range scan runs at parity because matching happens against the stored BSON
+bytes directly, skipping fields by length, with no per-document arena and no
+second Pair-tree copy. The log is still LZ4-compressed in 256 KiB blocks; since
+it is now truncated at every checkpoint, its size no longer tracks the
+database's.
+
+Reproduce the whole thing — main suite, concurrency sweep and the meta rows —
+with `bash tests/e2e/bench-run.sh 1g 16k "1 4 8 16 32"`, which writes a
+timestamped report to `tests/e2e/results/` and diffs it against the last one.
+The pre-tree baseline is in `tests/e2e/results/phase1.txt`; the runs with the
+B+tree, ordered `_id` index, compressed log and byte storage (roadmap items
+1–4) in `phase2.txt` through `phase5.txt`; the all-in-RAM engine this table's
+predecessor measured in `phase8.txt`; and the mmap foundation's own gate
+results, including what each of those rows cost or gained, in
+`m0-gates.txt`.
 
 ### What is left (highest impact first)