Aleksey Shakhmatov
3c1ab6f656
createIndex({expireAt: 1}, {expireAfterSeconds: 60}) now deletes a
document once its indexed date is that many seconds old.
index.zig carries the option: Index.ttl (?i64), parsed from
expireAfterSeconds (int32/int64/integral double, within MongoDB's
[0, 2147483647]; 0 means "expire at the stored instant"), emitted by
spec_pairs and so persisted through the log and reported by listIndexes,
and compared by spec_equal — a same-name re-create with a different
expiry stays IndexOptionsConflict, as in MongoDB, since collMod does not
exist here. The bound keeps the value an int32 on the wire and makes the
emission cast unconditional. TTL is single-field only (a compound key is
error.TtlOnCompoundIndex); an absent option parses to null, so every
existing log record reparses unchanged.
db.zig sweeps: Engine.ttl_sweep(now_ms) walks each TTL index's entries
and deletes through the ordinary remove path, so an expiry is logged and
fsynced like any other write and holds across a restart. Expired ids are
duped before removal — remove frees the docs-map key that Entry.id
aliases — then sorted and deduped, because one document can be expired
by several entries (an array of dates expires on its earliest member,
which multikey expansion gives for free) or by several TTL indexes.
Selecting entries is a type test rather than a range lookup: bson
compare order ranks datetime above null, numbers and strings, so a
datetime upper bound would also select every value of a lesser type. A
sweep that deleted something checks the compaction threshold, since a
TTL-only workload never reaches the one in upsert.
commands.zig maps the new spec errors: CannotCreateIndex (67) for a
compound key or an out-of-range expiry, and InvalidIndexSpecificationOption
(197) for an expiry on {_id: 1}, which the idempotent _id no-op would
otherwise swallow.
server.zig runs the monitor as a member of the connection group, so the
existing group.cancel tears it down; it sleeps first, then sweeps under
the write lock, and logs rather than dies on a sweep failure.
--ttl-sweep-secs sets the interval (default 60, 0 leaves it unspawned).
Expiry is coarse by design, as in MongoDB: a document stays visible
until the next sweep, and a non-date value at the indexed path never
expires. Unit tests cover the spec round-trip and every rejection, the
sweep (inclusive cutoff, string/missing/future values untouched, second
sweep a no-op) and its survival of a reopen, and two TTL indexes over
one collection. e2e4.js exercises it through the Node driver.
142 lines
7.2 KiB
Markdown
142 lines
7.2 KiB
Markdown
# mongo-light
|
|
|
|
A lightweight, embedded MongoDB-compatible document database written in
|
|
Zig 0.16. Like SQLite, it stores everything in a single file; unlike SQLite,
|
|
it speaks the MongoDB wire protocol, so real clients — `mongosh`, the Node.js
|
|
driver, PyMongo — connect over TCP and just work.
|
|
|
|
## Quick start
|
|
|
|
```sh
|
|
zig build # build the server
|
|
zig build test # run the unit test suite
|
|
|
|
zig-out/bin/mongo-light --port 27017 --db data.log
|
|
|
|
# in another terminal:
|
|
mongosh --port 27017
|
|
> db.users.insertOne({name: "alice", age: 30})
|
|
> db.users.find({age: {$gt: 25}}).toArray()
|
|
> db.users.updateOne({name: "alice"}, {$set: {vip: true}})
|
|
> db.users.deleteOne({name: "bob"})
|
|
```
|
|
|
|
## Features
|
|
|
|
- **Wire protocol**: OP_MSG (2013) plus legacy OP_QUERY/OP_REPLY (2004/2001)
|
|
for the driver handshake; hello/isMaster with `maxWireVersion: 8`, so
|
|
modern drivers (Node, Python, mongosh) connect without workarounds.
|
|
- **BSON**: full parse/serialize round-trip for all common types
|
|
(including binary, regex, timestamps, ObjectId), canonical MongoDB
|
|
comparison order for sorting and range queries.
|
|
- **CRUD**: `insert`, `find` (filter, sort, skip/limit, projection),
|
|
`update` (multi/upsert), `delete`, `findAndModify`, `count`,
|
|
`aggregate` (`$match`, `$sort`, `$skip`, `$limit`, `$project`, `$count`,
|
|
`$group` with `$sum`), plus `create`/`drop`/`listCollections`/
|
|
`listDatabases`/`dropDatabase`.
|
|
- **Query operators**: `$eq` `$ne` `$gt` `$gte` `$lt` `$lte` `$in` `$nin`
|
|
`$exists` `$regex` (hand-rolled engine: anchors, `.`, `* + ?`, character
|
|
classes, groups, alternation, `i`/`s` options) `$not` `$and` `$or` `$nor`
|
|
`$size` `$all` `$elemMatch`, with dot paths and array multikey semantics.
|
|
- **Secondary indexes**: `createIndex`/`listIndexes`/`dropIndex` via the
|
|
three driver commands, single-field and compound, with `unique`,
|
|
`sparse` and `expireAfterSeconds` (TTL) options, persisted in the log
|
|
and rebuilt on open (compaction
|
|
re-emits them). The query planner turns equality / `$in` / range
|
|
predicates into index lookups across `find`, `count`, `update`,
|
|
`delete`, `findAndModify`, and a leading `$match` in `aggregate`; every
|
|
candidate is re-checked against the full filter, so an index that
|
|
over-approximates is merely slow, never wrong.
|
|
- **Update operators**: `$set` `$unset` `$inc` `$push` (`$each`) `$pull`
|
|
`$rename`, with dot-path creation (including array indices).
|
|
- **Storage**: append-only record log (CRC32-checked, `fsync` per write,
|
|
torn-tail tolerant) with in-memory indexes rebuilt on open and automatic
|
|
compaction (rewrite + atomic rename when the log grows past 16 MB).
|
|
Killed mid-write (`kill -9`), the database recovers all committed writes;
|
|
the log and compaction both work with relative or absolute `--db` paths.
|
|
Records up to the announced 16 MB `maxBsonObjectSize` replay correctly.
|
|
- **Concurrency**: a writer-preferring read/write lock splits command
|
|
execution — reads (`find`, `count`, `aggregate`, `list*`) run concurrently
|
|
across connections, writes (CRUD, DDL) are exclusive and totally ordered,
|
|
and handshake/no-op commands run lock-free. The log append + `fsync` still
|
|
happen under the write lock, so the crash guarantees are unchanged. Fine
|
|
for light workloads.
|
|
|
|
## Layout
|
|
|
|
```
|
|
src/
|
|
bson.zig BSON parse/serialize, ObjectId, canonical comparison order
|
|
wire.zig OP_MSG/OP_QUERY framing, message + reply builders
|
|
commands.zig command dispatch (hello, CRUD, aggregate, admin, indexes)
|
|
server.zig TCP accept loop, per-connection handlers, TTL sweep monitor
|
|
db.zig in-memory engine: db → collection → _id → document maps
|
|
storage.zig append-only log: records, replay, CRC validation
|
|
query.zig filter matcher, regex engine, sort, projection
|
|
index.zig secondary indexes: entries, search, query planner
|
|
update.zig update operators with dot-path navigation
|
|
main.zig CLI: --port, --bind, --db, --ttl-sweep-secs
|
|
```
|
|
|
|
## Indexes
|
|
|
|
`collection.createIndex({field: 1})` works against every driver; the index
|
|
is persisted in the log, survives restarts and compaction, and is used by
|
|
the query planner to narrow scans.
|
|
|
|
- **Key patterns**: single-field and compound (up to 32 fields), each key
|
|
`1` or `-1`. Descending order is metadata (entries are always stored
|
|
value-ascending); the default index name is MongoDB's `a_1_b_-1`.
|
|
`createIndex({_id: 1})` is an idempotent no-op — the docs map is the
|
|
`_id_` index — and `dropIndex("_id_")` errors.
|
|
- **Options**: `unique` (a conflicting write fails with E11000 naming the
|
|
index; per-document entries are deduped first, so `{a: [1,1]}` is legal)
|
|
and `sparse` (documents missing an indexed field are skipped).
|
|
- **TTL**: `createIndex({expireAt: 1}, {expireAfterSeconds: 60})` deletes a
|
|
document once its indexed date is that many seconds old. A background
|
|
sweeper runs every `--ttl-sweep-secs` seconds (default 60, `0` disables
|
|
it) and deletes through the ordinary write path, so each expiry is logged
|
|
and fsynced and holds across a restart. As in MongoDB the option is
|
|
single-field only (a compound key is `CannotCreateIndex`, code 67),
|
|
`expireAfterSeconds` must be a whole number in `[0, 2147483647]` (`0`
|
|
means "expire at the stored instant"), a non-date value at the path never
|
|
expires, an array of dates expires on its earliest member, and expiry is
|
|
coarse: a document stays visible until the next sweep. Changing the
|
|
expiry of an existing index is `IndexOptionsConflict` — `collMod` is not
|
|
implemented.
|
|
- **Multikey**: an array at an indexed path is indexed as a whole *and*
|
|
element-wise, mirroring the query matcher exactly, so both
|
|
`{tags: "a"}` and `{tags: ["a","b"]}` hit the index. A compound index
|
|
over two array paths rejects the document with MongoDB's "cannot index
|
|
parallel arrays".
|
|
- **Planner**: picks the index covering the longest leading run of
|
|
equality/`$in` predicates (cartesian product capped at 100 lookups),
|
|
optionally with a range on the next key. Ranges with both bounds fall
|
|
back to a scan on multikey indexes (a doc with `{a: [1,2]}` can satisfy
|
|
`{a: {$gt: 5, $lt: 25}}` across two entries), and sparse indexes are
|
|
never used for `null`-valued predicates. The `_id_` fast path resolves
|
|
`{_id: ...}` through the docs map unless the value's compare class is
|
|
serialization-ambiguous (int32 1, int64 1, double 1.0 compare equal but
|
|
hash differently — those fall back to a scan, as do string/symbol/code).
|
|
|
|
v1 limits: no index-accelerated sort, no hashed/text/geo/partial indexes,
|
|
and entry insert/removal is O(n) (a sorted array) — fine for a light
|
|
database, with a B-tree or id→entry map as the follow-up.
|
|
|
|
## Not (yet) implemented
|
|
|
|
- Authentication (SCRAM) — run without credentials
|
|
- Real cursors (all results are returned in one batch, cursor id 0)
|
|
- Transactions, change streams, replicasets
|
|
- Compression (OP_COMPRESSED)
|
|
- `dropCollection`/`dropDatabase` write no log record, so a dropped
|
|
collection (and its index definitions) resurrect on restart; and
|
|
compaction never resets `log_bytes`, so every write after the first
|
|
compaction re-triggers the threshold check
|
|
|
|
## Code style
|
|
|
|
Zig 0.16 idioms (`std.Io` threaded through everything, unmanaged
|
|
containers); user-declared functions use `snake_case` per this repo's house
|
|
style.
|