Amendment A5 and the `[M1.1]`/`[M1.2]` blocks. What they record is a result
with two halves, and the value is in keeping both:
The mechanism works. 934 MB reclaimed over the update run, occupancy at
1.06-1.26x its live data, and the counters that say so are in `serverStatus`
rather than inferred.
The ratio did not move. 1.94x delete-heavy and 2.46x update-heavy, identical
to the end-of-Stage-2 binary measured with the same harness. `file / live` is
a high-water mark because the data file never shrinks, and the mark is set in
the first round by the one thing reclamation cannot avoid: a rebuild needs a
whole second copy of the live data before the first can be freed. So ~2x is
the floor of a rebuild-based design and no threshold reaches it -- rebuilding
earlier lowers the garbage term and nothing else, rebuilding later raises it.
The plan said in advance what to do if this happened, which was to write it
down rather than tune, and to name incremental compaction through a
doc-id-to-offset indirection layer as the successor. Recorded, with its cost:
a second copy-on-write B+tree per collection, a second random read on point
lookup, and it undoes A3.
A second lever is named that the plan had not: 52% of the steady-state file is
space the database owns and is not using, so returning it to the filesystem is
worth more here than reclaiming harder. It needs the file never to shrink
below what the fallback generation references, which is its own crash-safety
pass.
D7.4's 1.65x for the delete line is corrected to 1.94x, and the correction is
the harness rather than a regression -- the same 1.94x comes out of the binary
that predates any of this work. The old ad-hoc version sampled ids to delete
blindly, which re-picks dead ones, so it deleted fewer documents than it
inserted and measured a collection that was quietly growing. The update line
reproduces D7.4 exactly, 2.46 against 2.47.
200-byte documents reclaim nothing, exactly as forecast, and the forecast
being written down beforehand is what makes that a result instead of a
disappointment. 3.93x on both binaries. Also noted, because the number invites
misreading: at that document size the two index trees are comparable to the
documents themselves and the file is already 2.18x before any churn -- index
structure, not slab garbage.
The gate results file said the replacement-style-update gap was found and not
fixed, and PLAN said it was left alone. Both were true when written and are not
now, so a reader would take the M0 scorecard for the current one. The M0 figures
stay as measured -- they are the gate result -- with a pointer to
tests/spec/scorecard.txt, which always holds the current number.
PLAN D7's six items, with the numbers and the command that reproduces each in
tests/e2e/results/m0-gates.txt. Unit tests green in both optimize modes, the
whole e2e matrix green, the spec scorecard byte-identical at 131/161/195, and
the large smoke run at the scale D7.3 asked for:
21.47 GB collection (1,310,720 x 16 KiB)
data file 21.75 GB (+1.3% over the documents)
log after the load 2.5 MB (checkpoints reclaim it)
kill -9 then reopen 0.5 s (0.5 s at 4 GB too -- flat)
RSS after reopen 237 MB (1.1% of the data)
count after restart 1,310,720 last document byte-intact
acked writes after kill 200/200
That is the milestone's claim, measured: an open costs the working set rather
than the size of the database. Before M0 the same measurement was 523 MB
resident for a 512 MB database, because recovering each document's `_id` meant
reading every document at open.
Two gates need reading rather than a tick, and m0-gates.txt says so where a
reader would otherwise take a tick for granted.
The churn gate settles at 1.65x live data (delete-heavy) to 2.47x
(update-heavy), flat, above the ~1.3x amendment A2 hoped for. Rebuild-only
reclamation cannot reach that: it needs a whole second copy of the live data
before the first can be freed. The gate existed to decide whether doc-level
free lists are needed after M0, and that is the answer.
Benchmark parity holds for every read and latency row inside the run-to-run
spread, and bulk insert regresses 24% (732 -> 555 MB/s), reproducibly across
three runs. Risk 1 as written: document bytes now reach the disk uncompressed
on top of the LZ4 log. createIndex improves 62% from the same change.
Three measurement bugs fixed while running the gates, because each would have
put a false number in the README:
- `compare-run.sh` measured "db on disk" as `du` of the log alone against
`du` of mongod's whole dbpath. It reported 20 MB for a 1 GB collection --
the documents had moved to <db>.data. Honest figure, measured: 914 MB of
allocated blocks against mongod's compressed 85 MB.
- `big.js` counted "compaction events" as "the log shrank", which is a
*checkpoint* now. It claimed 12 compaction rewrites during a pure insert
load, which has no garbage to compact.
- `big.js` labelled peak RSS "in-memory engine: docs live in RAM" and its
summary said the collection was held "fully in RAM". Both were true of the
engine this milestone replaced.
README: the storage section described an all-in-RAM engine; the comparison
table mixed one old run's body with three new rows; and `findOne({_id})` was
documented as a full scan for integer ids, which the ordered `_id_` index made
false (2 ms against 55 s for a scan of the same 21.5 GB collection). The table
is now best-of-three for both servers, with the measured variance stated, since
two runs of the same binary moved the sub-10 ms rows by 27-51%.
