diff --git a/README.md b/README.md index feb0c8b..9aa52ef 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,13 @@ Zig 0.16. Like SQLite, it stores everything in a single file; unlike SQLite, it speaks the MongoDB wire protocol, so real clients — `mongosh`, the Node.js driver, PyMongo — connect over TCP and just work. +## Forward plan + +The direction from this MVP — a full-fledged embedded, tens-of-GB, +maximally MongoDB-compatible database — its decision record, milestones +and gates live in [PLAN.md](PLAN.md). Milestone 0 (mmap + WAL storage +foundation) is next. + ## Quick start ```sh @@ -317,6 +324,9 @@ Several real bugs surfaced while benchmarking: ## Code style -Zig 0.16 idioms (`std.Io` threaded through everything, unmanaged -containers); user-declared functions use `snake_case` per this repo's house -style. +The project follows TigerBeetle's TigerStyle — see +[`docs/TIGER_STYLE.md`](docs/TIGER_STYLE.md) (binding reference) and the +"Code style" section of `AGENTS.md` for the project-specific rules and +deliberate deviations. Highlights: `zig fmt` clean, 100-column hard limit, +4-space indent, snake_case, functions under 70 lines, always-on +assertions via `src/assert.zig`. diff --git a/docs/TIGER_STYLE.md b/docs/TIGER_STYLE.md new file mode 100644 index 0000000..e84ba5d --- /dev/null +++ b/docs/TIGER_STYLE.md @@ -0,0 +1,516 @@ + + +# TigerStyle + +## The Essence Of Style + +> “There are three things extremely hard: steel, a diamond, and to know one's self.” — Benjamin +> Franklin + +TigerBeetle's coding style is evolving. A collective give-and-take at the intersection of +engineering and art. Numbers and human intuition. Reason and experience. First principles and +knowledge. Precision and poetry. Just like music. A tight beat. A rare groove. Words that rhyme and +rhymes that break. Biodigital jazz. This is what we've learned along the way. The best is yet to +come. + +## Why Have Style? + +Another word for style is design. + +> “The design is not just what it looks like and feels like. The design is how it works.” — Steve +> Jobs + +Our design goals are safety, performance, and developer experience. In that order. All three are +important. Good style advances these goals. Does the code make for more or less safety, performance +or developer experience? That is why we need style. + +Put this way, style is more than readability, and readability is table stakes, a means to an end +rather than an end in itself. + +> “...in programming, style is not something to pursue directly. Style is necessary only where +> understanding is missing.” ─ [Let Over +> Lambda](https://letoverlambda.com/index.cl/guest/chap1.html) + +This document explores how we apply these design goals to coding style. First, a word on simplicity, +elegance and technical debt. + +## On Simplicity And Elegance + +Simplicity is not a free pass. It's not in conflict with our design goals. It need not be a +concession or a compromise. + +Rather, simplicity is how we bring our design goals together, how we identify the “super idea” that +solves the axes simultaneously, to achieve something elegant. + +> “Simplicity and elegance are unpopular because they require hard work and discipline to achieve” — +> Edsger Dijkstra + +Contrary to popular belief, simplicity is also not the first attempt but the hardest revision. It's +easy to say “let's do something simple”, but to do that in practice takes thought, multiple passes, +many sketches, and still we may have to [“throw one +away”](https://en.wikipedia.org/wiki/The_Mythical_Man-Month). + +The hardest part, then, is how much thought goes into everything. + +We spend this mental energy upfront, proactively rather than reactively, because we know that when +the thinking is done, what is spent on the design will be dwarfed by the implementation and testing, +and then again by the costs of operation and maintenance. + +An hour or day of design is worth weeks or months in production: + +> “the simple and elegant systems tend to be easier and faster to design and get right, more +> efficient in execution, and much more reliable” — Edsger Dijkstra + +## Technical Debt + +What could go wrong? What's wrong? Which question would we rather ask? The former, because code, +like steel, is less expensive to change while it's hot. A problem solved in production is many times +more expensive than a problem solved in implementation, or a problem solved in design. + +Since it's hard enough to discover showstoppers, when we do find them, we solve them. We don't allow +potential memcpy latency spikes, or exponential complexity algorithms to slip through. + +> “You shall not pass!” — Gandalf + +In other words, TigerBeetle has a “zero technical debt” policy. We do it right the first time. This +is important because the second time may not transpire, and because doing good work, that we can be +proud of, builds momentum. + +We know that what we ship is solid. We may lack crucial features, but what we have meets our design +goals. This is the only way to make steady incremental progress, knowing that the progress we have +made is indeed progress. + +## Safety + +> “The rules act like the seat-belt in your car: initially they are perhaps a little uncomfortable, +> but after a while their use becomes second-nature and not using them becomes unimaginable.” — +> Gerard J. Holzmann + +[NASA's Power of Ten — Rules for Developing Safety Critical +Code](https://spinroot.com/gerard/pdf/P10.pdf) will change the way you code forever. To expand: + +- Use **only very simple, explicit control flow** for clarity. **Do not use recursion** to ensure + that all executions that should be bounded are bounded. Use **only a minimum of excellent + abstractions** but only if they make the best sense of the domain. Abstractions are [never zero + cost](https://isaacfreund.com/blog/2022-05/). Every abstraction introduces the risk of a leaky + abstraction. + +- **Put a limit on everything** because, in reality, this is what we expect—everything has a limit. + For example, all loops and all queues must have a fixed upper bound to prevent infinite loops or + tail latency spikes. This follows the [“fail-fast”](https://en.wikipedia.org/wiki/Fail-fast) + principle so that violations are detected sooner rather than later. Where a loop cannot terminate + (e.g. an event loop), this must be asserted. + +- Use explicitly-sized types like `u32` for everything, avoid architecture-specific `usize`. + +- **Assertions detect programmer errors. Unlike operating errors, which are expected and which must + be handled, assertion failures are unexpected. The only correct way to handle corrupt code is to + crash. Assertions downgrade catastrophic correctness bugs into liveness bugs. Assertions are a + force multiplier for discovering bugs by fuzzing.** + + - **Assert all function arguments and return values, pre/postconditions and invariants.** A + function must not operate blindly on data it has not checked. The purpose of a function is to + increase the probability that a program is correct. Assertions within a function are part of how + functions serve this purpose. The assertion density of the code must average a minimum of two + assertions per function. + + - **[Pair assertions](https://tigerbeetle.com/blog/2023-12-27-it-takes-two-to-contract).** For + every property you want to enforce, try to find at least two different code paths where an + assertion can be added. For example, assert validity of data right before writing it to disk, + and also immediately after reading from disk. + + - On occasion, you may use a blatantly true assertion instead of a comment as stronger + documentation where the assertion condition is critical and surprising. + + - Split compound assertions: prefer `assert(a); assert(b);` over `assert(a and b);`. + The former is simpler to read, and provides more precise information if the condition fails. + + - Use single-line `if` to assert an implication: `if (a) assert(b)`. + + - **Assert the relationships of compile-time constants** as a sanity check, and also to document + and enforce [subtle + invariants](https://github.com/coilhq/tigerbeetle/blob/db789acfb93584e5cb9f331f9d6092ef90b53ea6/src/vsr/journal.zig#L45-L47) + or [type + sizes](https://github.com/coilhq/tigerbeetle/blob/578ac603326e1d3d33532701cb9285d5d2532fe7/src/ewah.zig#L41-L53). + Compile-time assertions are extremely powerful because they are able to check a program's design + integrity _before_ the program even executes. + + - **The golden rule of assertions is to assert the _positive space_ that you do expect AND to + assert the _negative space_ that you do not expect** because where data moves across the + valid/invalid boundary between these spaces is where interesting bugs are often found. This is + also why **tests must test exhaustively**, not only with valid data but also with invalid data, + and as valid data becomes invalid. + + - Assertions are a safety net, not a substitute for human understanding. With simulation testing, + there is the temptation to trust the fuzzer. But a fuzzer can prove only the presence of bugs, + not their absence. Therefore: + - Build a precise mental model of the code first, + - encode your understanding in the form of assertions, + - write the code and comments to explain and justify the mental model to your reviewer, + - and use VOPR as the final line of defense, to find bugs in your and reviewer's understanding + of code. + +- All memory must be statically allocated at startup. **No memory may be dynamically allocated (or + freed and reallocated) after initialization.** This avoids unpredictable behavior that can + significantly affect performance, and avoids use-after-free. As a second-order effect, it is our + experience that this also makes for more efficient, simpler designs that are more performant and + easier to maintain and reason about, compared to designs that do not consider all possible memory + usage patterns upfront as part of the design. + +- Declare variables at the **smallest possible scope**, and **minimize the number of variables in + scope**, to reduce the probability that variables are misused. + +- There's a sharp discontinuity between a function fitting on a screen, and having to scroll to + see how long it is. For this physical reason we enforce a **hard limit of 70 lines per function**. + Art is born of constraints. There are many ways to cut a wall of code into chunks of 70 lines, + but only a few splits will feel right. Some rules of thumb: + + * Good function shape is often the inverse of an hourglass: a few parameters, a simple return + type, and a lot of meaty logic between the braces. + * Centralize control flow. When splitting a large function, try to keep all switch/if + statements in the "parent" function, and move non-branchy logic fragments to helper + functions. Divide responsibility. All control flow should be handled by _one_ function, the rest shouldn't + care about control flow at all. In other words, + ["push `if`s up and `for`s down"](https://matklad.github.io/2023/11/15/push-ifs-up-and-fors-down.html). + * Similarly, centralize state manipulation. Let the parent function keep all relevant state in + local variables, and use helpers to compute what needs to change, rather than applying the + change directly. Keep leaf functions pure. + +- Appreciate, from day one, **all compiler warnings at the compiler's strictest setting**. + +- Whenever your program has to interact with external entities, **don't do things directly in + reaction to external events**. Instead, your program should run at its own pace. Not only does + this make your program safer by keeping the control flow of your program under your control, it + also improves performance for the same reason (you get to batch, instead of context switching on + every event). Additionally, this makes it easier to maintain bounds on work done per time period. + +Beyond these rules: + +- Compound conditions that evaluate multiple booleans make it difficult for the reader to verify + that all cases are handled. Split compound conditions into simple conditions using nested + `if/else` branches. Split complex `else if` chains into `else { if { } }` trees. This makes the + branches and cases clear. Again, consider whether a single `if` does not also need a matching + `else` branch, to ensure that the positive and negative spaces are handled or asserted. + +- Negations are not easy! State invariants positively. When working with lengths and indexes, this + form is easy to get right (and understand): + + ```zig + if (index < length) { + // The invariant holds. + } else { + // The invariant doesn't hold. + } + ``` + + This form is harder, and also goes against the grain of how `index` would typically be compared to + `length`, for example, in a loop condition: + + ```zig + if (index >= length) { + // It's not true that the invariant holds. + } + ``` + +- All errors must be handled. An [analysis of production failures in distributed data-intensive + systems](https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-yuan.pdf) found that + the majority of catastrophic failures could have been prevented by simple testing of error + handling code. + +> “Specifically, we found that almost all (92%) of the catastrophic system failures are the result +> of incorrect handling of non-fatal errors explicitly signaled in software.” + +- **Always motivate, always say why**. Never forget to say why. Because if you explain the rationale + for a decision, it not only increases the hearer's understanding, and makes them more likely to + adhere or comply, but it also shares criteria with them with which to evaluate the decision and + its importance. + +- **Explicitly pass options to library functions at the call site, instead of relying on the + defaults**. For example, write `@prefetch(a, .{ .cache = .data, .rw = .read, .locality = 3 });` + over `@prefetch(a, .{});`. This improves readability but most of all avoids latent, potentially + catastrophic bugs in case the library ever changes its defaults. + +## Performance + +> “The lack of back-of-the-envelope performance sketches is the root of all evil.” — Rivacindela +> Hudsoni + +- Think about performance from the outset, from the beginning. **The best time to solve performance, + to get the huge 1000x wins, is in the design phase, which is precisely when we can't measure or + profile.** It's also typically harder to fix a system after implementation and profiling, and the + gains are less. So you have to have mechanical sympathy. Like a carpenter, work with the grain. + +- **Perform back-of-the-envelope sketches with respect to the four resources (network, disk, memory, + CPU) and their two main characteristics (bandwidth, latency).** Sketches are cheap. Use sketches + to be “roughly right” and land within 90% of the global maximum. + +- Optimize for the slowest resources first (network, disk, memory, CPU) in that order, after + compensating for the frequency of usage, because faster resources may be used many times more. For + example, a memory cache miss may be as expensive as a disk fsync, if it happens many times more. + +- Distinguish between the control plane and data plane. A clear delineation between control plane + and data plane through the use of batching enables a high level of assertion safety without losing + performance. See our [July 2021 talk on Zig SHOWTIME](https://youtu.be/BH2jvJ74npM?t=1958) for + examples. + +- Amortize network, disk, memory and CPU costs by batching accesses. + +- Let the CPU be a sprinter doing the 100m. Be predictable. Don't force the CPU to zig zag and + change lanes. Give the CPU large enough chunks of work. This comes back to batching. + +- Be explicit. Minimize dependence on the compiler to do the right thing for you. + + In particular, extract hot loops into stand-alone functions with primitive arguments without + `self` (see [an example](https://github.com/tigerbeetle/tigerbeetle/blob/0.16.19/src/lsm/compaction.zig#L1932-L1937)). + That way, the compiler doesn't need to prove that it can cache struct's fields in registers, and a + human reader can spot redundant computations easier. + +## Developer Experience + +> “There are only two hard things in Computer Science: cache invalidation, naming things, and +> off-by-one errors.” — Phil Karlton + +### Naming Things + +- **Get the nouns and verbs just right.** Great names are the essence of great code, they capture + what a thing is or does, and provide a crisp, intuitive mental model. They show that you + understand the domain. Take time to find the perfect name, to find nouns and verbs that work + together, so that the whole is greater than the sum of its parts. + +- Use `snake_case` for function, variable, and file names. The underscore is the closest thing we + have as programmers to a space, and helps to separate words and encourage descriptive names. We + don't use Zig's `CamelCase.zig` style for "struct" files to keep the convention simple and + consistent. + +- Do not abbreviate variable names, unless the variable is a primitive integer type used as an + argument to a sort function or matrix calculation. Use long form arguments in scripts: `--force`, + not `-f`. Single letter flags are for interactive usage. + +- Use proper capitalization for acronyms (`VSRState`, not `VsrState`). + +- For the rest, follow the Zig style guide. + +- Add units or qualifiers to variable names, and put the units or qualifiers last, sorted by + descending significance, so that the variable starts with the most significant word, and ends with + the least significant word. For example, `latency_ms_max` rather than `max_latency_ms`. This will + then line up nicely when `latency_ms_min` is added, as well as group all variables that relate to + latency. + +- Infuse names with meaning. For example, `allocator: Allocator` is a good, if boring name, + but `gpa: Allocator` and `arena: Allocator` are excellent. They inform the reader whether + `deinit` should be called explicitly. + +- When choosing related names, try hard to find names with the same number of characters so that + related variables all line up in the source. For example, as arguments to a memcpy function, + `source` and `target` are better than `src` and `dest` because they have the second-order effect + that any related variables such as `source_offset` and `target_offset` will all line up in + calculations and slices. This makes the code symmetrical, with clean blocks that are easier for + the eye to parse and for the reader to check. + +- When a single function calls out to a helper function or callback, prefix the name of the helper + function with the name of the calling function to show the call history. For example, + `read_sector()` and `read_sector_callback()`. + +- Callbacks go last in the list of parameters. This mirrors control flow: callbacks are also + _invoked_ last. + +- _Order_ matters for readability (even if it doesn't affect semantics). On the first read, a file + is read top-down, so put important things near the top. The `main` function goes first. + + The same goes for `structs`, the order is fields then types then methods: + + ```zig + time: Time, + process_id: ProcessID, + + const ProcessID = struct { cluster: u128, replica: u8 }; + const Tracer = @This(); // This alias concludes the types section. + + pub fn init(gpa: std.mem.Allocator, time: Time) !Tracer { + ... + } + ``` + + If a nested type is complex, make it a top-level struct. + + At the same time, not everything has a single right order. When in doubt, consider sorting + alphabetically, taking advantage of big-endian naming. + +- Don't overload names with multiple meanings that are context-dependent. For example, TigerBeetle + has a feature called _pending transfers_ where a pending transfer can be subsequently _posted_ or + _voided_. At first, we called them _two-phase commit transfers_, but this overloaded the + _two-phase commit_ terminology that was used in our consensus protocol, causing confusion. + +- Think of how names will be used outside the code, in documentation or communication. For example, + a noun is often a better descriptor than an adjective or present participle, because a noun can be + directly used in correspondence without having to be rephrased. Compare `replica.pipeline` vs + `replica.preparing`. The former can be used directly as a section header in a document or + conversation, whereas the latter must be clarified. Noun names compose more clearly for derived + identifiers, e.g. `config.pipeline_max`. + +- Zig has named arguments through the `options: struct` pattern. Use it when arguments can be + mixed up. A function taking two `u64` must use an options struct. If an argument can be `null`, + it should be named so that the meaning of `null` literal at the call site is clear. + + Because dependencies like an allocator or a tracer are singletons with unique types, they should + be threaded through constructors positionally, from the most general to the most specific. + +- **Write descriptive commit messages** that inform and delight the reader, because your commit + messages are being read. Note that a pull request description is not stored in the git repository + and is invisible in `git blame`, and therefore is not a replacement for a commit message. + +- Don't forget to say why. Code alone is not documentation. Use comments to explain why you wrote + the code the way you did. Show your workings. + +- Don't forget to say how. For example, when writing a test, think of writing a description at the + top to explain the goal and methodology of the test, to help your reader get up to speed, or to + skip over sections, without forcing them to dive in. + +- Comments are sentences, with a space after the slash, with a capital letter and a full stop, or a + colon if they relate to something that follows. Comments are well-written prose describing the + code, not just scribblings in the margin. Comments after the end of a line _can_ be phrases, with + no punctuation. + +### Cache Invalidation + +- Don't duplicate variables or take aliases to them. This will reduce the probability that state + gets out of sync. + +- If you don't mean a function argument to be copied when passed by value, and if the argument type + is more than 16 bytes, then pass the argument as `*const`. This will catch bugs where the caller + makes an accidental copy on the stack before calling the function. + +- Construct larger structs _in-place_ by passing an _out pointer_ during initialization. + + In-place initializations can assume **pointer stability** and **immovable types** while + eliminating intermediate copy-move allocations, which can lead to undesirable stack growth. + + Keep in mind that in-place initializations are viral — if any field is initialized + in-place, the entire container struct should be initialized in-place as well. + + **Prefer:** + ```zig + fn init(target: *LargeStruct) !void { + target.* = .{ + // in-place initialization. + }; + } + + fn main() !void { + var target: LargeStruct = undefined; + try target.init(); + } + ``` + + **Over:** + ```zig + fn init() !LargeStruct { + return LargeStruct { + // moving the initialized object. + } + } + + fn main() !void { + var target = try LargeStruct.init(); + } + ``` + +- **Shrink the scope** to minimize the number of variables at play and reduce the probability that + the wrong variable is used. + +- Calculate or check variables close to where/when they are used. **Don't introduce variables before + they are needed.** Don't leave them around where they are not. This will reduce the probability of + a POCPOU (place-of-check to place-of-use), a distant cousin to the infamous + [TOCTOU](https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use). Most bugs come down to a + semantic gap, caused by a gap in time or space, because it's harder to check code that's not + contained along those dimensions. + +- Use simpler function signatures and return types to reduce dimensionality at the call site, the + number of branches that need to be handled at the call site, because this dimensionality can also + be viral, propagating through the call chain. For example, as a return type, `void` trumps `bool`, + `bool` trumps `u64`, `u64` trumps `?u64`, and `?u64` trumps `!u64`. + +- Ensure that functions run to completion without suspending, so that precondition assertions are + true throughout the lifetime of the function. These assertions are useful documentation without a + suspend, but may be misleading otherwise. + +- Be on your guard for **[buffer bleeds](https://en.wikipedia.org/wiki/Heartbleed)**. This is a + buffer underflow, the opposite of a buffer overflow, where a buffer is not fully utilized, with + padding not zeroed correctly. This may not only leak sensitive information, but may cause + deterministic guarantees as required by TigerBeetle to be violated. + +- Use newlines to **group resource allocation and deallocation**, i.e. before the resource + allocation and after the corresponding `defer` statement, to make leaks easier to spot. + +### Off-By-One Errors + +- **The usual suspects for off-by-one errors are casual interactions between an `index`, a `count` + or a `size`.** These are all primitive integer types, but should be seen as distinct types, with + clear rules to cast between them. To go from an `index` to a `count` you need to add one, since + indexes are _0-based_ but counts are _1-based_. To go from a `count` to a `size` you need to + multiply by the unit. Again, this is why including units and qualifiers in variable names is + important. + +- Show your intent with respect to division. For example, use `@divExact()`, `@divFloor()` or + `div_ceil()` to show the reader you've thought through all the interesting scenarios where + rounding may be involved. + +### Style By The Numbers + +- Run `zig fmt`. + +- Use 4 spaces of indentation, rather than 2 spaces, as that is more obvious to the eye at a + distance. + +- Hard limit all line lengths, without exception, to at most 100 columns for a good typographic + "measure". Use it up. Never go beyond. Nothing should be hidden by a horizontal scrollbar. Let + your editor help you by setting a column ruler. To wrap a function signature, call or data + structure, add a trailing comma, close your eyes and let `zig fmt` do the rest. + + Similar to function length, the motivation behind the number 100 is physical: just enough + to fit two copies of the code side-by-side on a screen. + +- Add braces to the `if` statement unless it fits on a single line for consistency and defense in + depth against "goto fail;" bugs. + +### Dependencies + +TigerBeetle has **a “zero dependencies” policy**, apart from the Zig toolchain. Dependencies, in +general, inevitably lead to supply chain attacks, safety and performance risk, and slow install +times. For foundational infrastructure in particular, the cost of any dependency is further +amplified throughout the rest of the stack. + +### Tooling + +Similarly, tools have costs. A small standardized toolbox is simpler to operate than an array of +specialized instruments each with a dedicated manual. Our primary tool is Zig. It may not be the +best for everything, but it's good enough for most things. We invest into our Zig tooling to ensure +that we can tackle new problems quickly, with a minimum of accidental complexity in our local +development environment. + +> “The right tool for the job is often the tool you are already using—adding new tools has a higher +> cost than many people appreciate” — John Carmack + +For example, the next time you write a script, instead of `scripts/*.sh`, write `scripts/*.zig`. + +This not only makes your script cross-platform and portable, but introduces type safety and +increases the probability that running your script will succeed for everyone on the team, instead of +hitting a Bash/Shell/OS-specific issue. + +Standardizing on Zig for tooling is important to ensure that we reduce dimensionality, as the team, +and therefore the range of personal tastes, grows. This may be slower for you in the short term, but +makes for more velocity for the team in the long term. + +## The Last Stage + +At the end of the day, keep trying things out, have fun, and remember—it's called TigerBeetle, not +only because it's fast, but because it's small! + +> You don’t really suppose, do you, that all your adventures and escapes were managed by mere luck, +> just for your sole benefit? You are a very fine person, Mr. Baggins, and I am very fond of you; +> but you are only quite a little fellow in a wide world after all!” +> +> “Thank goodness!” said Bilbo laughing, and handed him the tobacco-jar. diff --git a/src/bson.zig b/src/bson.zig index 8ae16a3..5dfa35e 100644 --- a/src/bson.zig +++ b/src/bson.zig @@ -122,7 +122,11 @@ pub const Document = struct { } /// Serialize the full document (length-prefixed) into `out`. - pub fn to_bytes(self: *const Document, gpa: std.mem.Allocator, out: *std.ArrayListUnmanaged(u8)) !void { + pub fn to_bytes( + self: *const Document, + gpa: std.mem.Allocator, + out: *std.ArrayListUnmanaged(u8), + ) !void { try write_doc(self.pairs, gpa, out); } }; @@ -157,7 +161,12 @@ const Parser = struct { const ParseError = error{ InvalidBson, OutOfMemory }; -fn parse_doc_into(allocator: std.mem.Allocator, bytes: []const u8, idx: *usize, borrow: bool) ParseError![]const Pair { +fn parse_doc_into( + allocator: std.mem.Allocator, + bytes: []const u8, + idx: *usize, + borrow: bool, +) ParseError![]const Pair { const p = Parser{ .allocator = allocator, .bytes = bytes, .borrow = borrow }; return parse_doc_inner(p, idx); } @@ -356,7 +365,10 @@ fn parse_array(p: Parser, idx: *usize) ParseError![]const Value { /// A borrowed parse of `bytes` into an arena: keys, strings, binary and /// regex payloads point into `bytes`; only the pair/value skeleton is /// allocated. The result is valid while both `bytes` and `arena` live. -pub fn spine(allocator: std.mem.Allocator, bytes: []const u8) error{ InvalidBson, OutOfMemory }![]const Pair { +pub fn spine( + allocator: std.mem.Allocator, + bytes: []const u8, +) error{ InvalidBson, OutOfMemory }![]const Pair { var idx: usize = 0; return parse_doc_into(allocator, bytes, &idx, true); } @@ -414,7 +426,12 @@ pub fn skip_value(bytes: []const u8, idx: *usize, tag: u8) error{InvalidBson}!vo /// Read one value of `tag` into a Value whose leaves borrow `bytes`; nested /// documents and arrays materialize their spines into `arena`. Advances /// `idx` past the value. -pub fn read_value(allocator: std.mem.Allocator, bytes: []const u8, idx: *usize, tag: u8) error{ InvalidBson, OutOfMemory }!Value { +pub fn read_value( + allocator: std.mem.Allocator, + bytes: []const u8, + idx: *usize, + tag: u8, +) error{ InvalidBson, OutOfMemory }!Value { switch (tag) { 0x01 => { try ensure_available(bytes, idx.*, 8); @@ -542,7 +559,11 @@ pub fn read_value(allocator: std.mem.Allocator, bytes: []const u8, idx: *usize, /// The value stored under `key` in a document's bytes, or null when absent. /// Nested documents and arrays materialize their spines into `arena`. -pub fn get_at(arena: std.mem.Allocator, bytes: []const u8, key: []const u8) error{ InvalidBson, OutOfMemory }!?Value { +pub fn get_at( + arena: std.mem.Allocator, + bytes: []const u8, + key: []const u8, +) error{ InvalidBson, OutOfMemory }!?Value { var idx: usize = 4; while (idx + 1 < bytes.len and bytes[idx] != 0) { const tag = bytes[idx]; @@ -565,7 +586,11 @@ pub const SerializeError = error{ OutOfMemory, }; -pub fn write_value(v: Value, gpa: std.mem.Allocator, out: *std.ArrayListUnmanaged(u8)) SerializeError!void { +pub fn write_value( + v: Value, + gpa: std.mem.Allocator, + out: *std.ArrayListUnmanaged(u8), +) SerializeError!void { switch (v) { .double => |d| { var buf: [8]u8 = undefined; @@ -618,13 +643,21 @@ pub fn write_value(v: Value, gpa: std.mem.Allocator, out: *std.ArrayListUnmanage } } -pub fn write_cstring(s: []const u8, gpa: std.mem.Allocator, out: *std.ArrayListUnmanaged(u8)) SerializeError!void { +pub fn write_cstring( + s: []const u8, + gpa: std.mem.Allocator, + out: *std.ArrayListUnmanaged(u8), +) SerializeError!void { if (std.mem.indexOfScalar(u8, s, 0) != null) return error.BsonNulInKey; try out.appendSlice(gpa, s); try out.append(gpa, 0); } -pub fn write_string(s: []const u8, gpa: std.mem.Allocator, out: *std.ArrayListUnmanaged(u8)) SerializeError!void { +pub fn write_string( + s: []const u8, + gpa: std.mem.Allocator, + out: *std.ArrayListUnmanaged(u8), +) SerializeError!void { if (s.len + 1 > std.math.maxInt(u32)) return error.BsonTooLarge; var buf: [4]u8 = undefined; std.mem.writeInt(u32, &buf, @intCast(s.len + 1), .little); @@ -633,7 +666,11 @@ pub fn write_string(s: []const u8, gpa: std.mem.Allocator, out: *std.ArrayListUn try out.append(gpa, 0); } -pub fn write_element(pair: Pair, gpa: std.mem.Allocator, out: *std.ArrayListUnmanaged(u8)) SerializeError!void { +pub fn write_element( + pair: Pair, + gpa: std.mem.Allocator, + out: *std.ArrayListUnmanaged(u8), +) SerializeError!void { try out.append(gpa, pair.value.type_tag()); try write_cstring(pair.key, gpa, out); try write_value(pair.value, gpa, out); @@ -648,7 +685,11 @@ fn begin_frame(gpa: std.mem.Allocator, out: *std.ArrayListUnmanaged(u8)) Seriali } /// Terminate the frame opened at `len_pos` and patch in its total length. -fn end_frame(gpa: std.mem.Allocator, out: *std.ArrayListUnmanaged(u8), len_pos: usize) SerializeError!void { +fn end_frame( + gpa: std.mem.Allocator, + out: *std.ArrayListUnmanaged(u8), + len_pos: usize, +) SerializeError!void { try out.append(gpa, 0); const total = out.items.len - len_pos; if (total > std.math.maxInt(u32)) return error.BsonTooLarge; @@ -656,13 +697,21 @@ fn end_frame(gpa: std.mem.Allocator, out: *std.ArrayListUnmanaged(u8), len_pos: } /// Write a length-prefixed document. Length is patched in after the body. -pub fn write_doc(pairs: []const Pair, gpa: std.mem.Allocator, out: *std.ArrayListUnmanaged(u8)) SerializeError!void { +pub fn write_doc( + pairs: []const Pair, + gpa: std.mem.Allocator, + out: *std.ArrayListUnmanaged(u8), +) SerializeError!void { const len_pos = try begin_frame(gpa, out); for (pairs) |p| try write_element(p, gpa, out); try end_frame(gpa, out, len_pos); } -fn write_array(items: []const Value, gpa: std.mem.Allocator, out: *std.ArrayListUnmanaged(u8)) SerializeError!void { +fn write_array( + items: []const Value, + gpa: std.mem.Allocator, + out: *std.ArrayListUnmanaged(u8), +) SerializeError!void { const len_pos = try begin_frame(gpa, out); var buf: [16]u8 = undefined; for (items, 0..) |item, i| { @@ -686,7 +735,11 @@ pub fn serialize_value(gpa: std.mem.Allocator, v: Value) ![]u8 { /// Append the serialized-key bytes of `v` (type tag + payload) to `out`. The /// appending form of serialize_value, for callers reusing one scratch buffer /// across many keys. -pub fn write_serialized_value(v: Value, gpa: std.mem.Allocator, out: *std.ArrayListUnmanaged(u8)) !void { +pub fn write_serialized_value( + v: Value, + gpa: std.mem.Allocator, + out: *std.ArrayListUnmanaged(u8), +) !void { try out.append(gpa, v.type_tag()); try write_value(v, gpa, out); } @@ -709,7 +762,10 @@ pub fn copy_value(arena: std.mem.Allocator, v: Value) std.mem.Allocator.Error!Va }; } -pub fn copy_pairs(arena: std.mem.Allocator, pairs: []const Pair) std.mem.Allocator.Error![]const Pair { +pub fn copy_pairs( + arena: std.mem.Allocator, + pairs: []const Pair, +) std.mem.Allocator.Error![]const Pair { const out = try arena.alloc(Pair, pairs.len); for (pairs, 0..) |p, i| { out[i] = .{ .key = try arena.dupe(u8, p.key), .value = try copy_value(arena, p.value) }; @@ -717,7 +773,10 @@ pub fn copy_pairs(arena: std.mem.Allocator, pairs: []const Pair) std.mem.Allocat return out; } -fn copy_values(arena: std.mem.Allocator, items: []const Value) std.mem.Allocator.Error![]const Value { +fn copy_values( + arena: std.mem.Allocator, + items: []const Value, +) std.mem.Allocator.Error![]const Value { const out = try arena.alloc(Value, items.len); for (items, 0..) |item, i| out[i] = try copy_value(arena, item); return out; @@ -879,7 +938,11 @@ pub fn encoded_leading_datetime(key: []const u8) ?i64 { /// ambiguous. Escaping `00` as `00 FF` fixes both problems at once: a real /// NUL encodes above the `00 00` terminator, and any byte >= 01 is above it /// too, so "shorter is less" falls out to match `std.mem.order`. -fn encode_escaped(bytes: []const u8, gpa: std.mem.Allocator, out: *std.ArrayListUnmanaged(u8)) !void { +fn encode_escaped( + bytes: []const u8, + gpa: std.mem.Allocator, + out: *std.ArrayListUnmanaged(u8), +) !void { for (bytes) |b| { try out.append(gpa, b); if (b == 0x00) try out.append(gpa, 0xFF); @@ -1128,23 +1191,47 @@ test "encode_key order matches bson.compare on every pair" { .min_key, .null, // Numbers: cross-type equality, sign, zero, extremes, NaN. - .{ .int32 = -2147483648 }, .{ .int32 = -1 }, .{ .int32 = 0 }, - .{ .int32 = 1 }, .{ .int32 = 2 }, .{ .int32 = 2147483647 }, - .{ .int64 = -9223372036854775807 }, .{ .int64 = -1 }, .{ .int64 = 0 }, - .{ .int64 = 1 }, .{ .int64 = 9223372036854775807 }, - .{ .double = -std.math.inf(f64) }, .{ .double = -1.5 }, .{ .double = -0.0 }, - .{ .double = 0.0 }, .{ .double = 0.5 }, .{ .double = 1.0 }, - .{ .double = 1.5 }, .{ .double = std.math.inf(f64) }, + .{ .int32 = -2147483648 }, + .{ .int32 = -1 }, + .{ .int32 = 0 }, + .{ .int32 = 1 }, + .{ .int32 = 2 }, + .{ .int32 = 2147483647 }, + .{ .int64 = -9223372036854775807 }, + .{ .int64 = -1 }, + .{ .int64 = 0 }, + .{ .int64 = 1 }, + .{ .int64 = 9223372036854775807 }, + .{ .double = -std.math.inf(f64) }, + .{ .double = -1.5 }, + .{ .double = -0.0 }, + .{ .double = 0.0 }, + .{ .double = 0.5 }, + .{ .double = 1.0 }, + .{ .double = 1.5 }, + .{ .double = std.math.inf(f64) }, .{ .double = std.math.nan(f64) }, // Strings, including embedded NUL and prefix relationships. - .{ .string = "" }, .{ .string = "\x00" }, .{ .string = "\x00b" }, - .{ .string = "a" }, .{ .string = "a\x00" }, .{ .string = "a\x00b" }, - .{ .string = "ab" }, .{ .string = "b" }, .{ .string = "\xff" }, + .{ .string = "" }, + .{ .string = "\x00" }, + .{ .string = "\x00b" }, + .{ .string = "a" }, + .{ .string = "a\x00" }, + .{ .string = "a\x00b" }, + .{ .string = "ab" }, + .{ .string = "b" }, + .{ .string = "\xff" }, // Same rank as string, so these must interleave with them. - .{ .symbol = "a" }, .{ .code = "ab" }, - .{ .doc = &.{} }, .{ .doc = &nested }, .{ .doc = &nested2 }, - .{ .doc = &nested_l }, .{ .doc = &two_pairs }, - .{ .array = &.{} }, .{ .array = &arr1 }, .{ .array = &arr2 }, + .{ .symbol = "a" }, + .{ .code = "ab" }, + .{ .doc = &.{} }, + .{ .doc = &nested }, + .{ .doc = &nested2 }, + .{ .doc = &nested_l }, + .{ .doc = &two_pairs }, + .{ .array = &.{} }, + .{ .array = &arr1 }, + .{ .array = &arr2 }, .{ .array = &arr_str }, // Binary orders by length first, then bytes, then subtype. .{ .binary = .{ .subtype = 0, .data = "" } }, @@ -1155,11 +1242,15 @@ test "encode_key order matches bson.compare on every pair" { .{ .object_id = [_]u8{0} ** 12 }, .{ .object_id = [_]u8{0} ** 11 ++ [_]u8{1} }, .{ .object_id = [_]u8{255} ** 12 }, - .{ .bool = false }, .{ .bool = true }, - .{ .datetime = std.math.minInt(i64) }, .{ .datetime = -1 }, - .{ .datetime = 0 }, .{ .datetime = 1 }, + .{ .bool = false }, + .{ .bool = true }, + .{ .datetime = std.math.minInt(i64) }, + .{ .datetime = -1 }, + .{ .datetime = 0 }, + .{ .datetime = 1 }, .{ .datetime = std.math.maxInt(i64) }, - .{ .timestamp = 0 }, .{ .timestamp = 1 }, + .{ .timestamp = 0 }, + .{ .timestamp = 1 }, .{ .timestamp = std.math.maxInt(u64) }, .{ .regex = .{ .pattern = "a", .options = "" } }, .{ .regex = .{ .pattern = "a", .options = "i" } }, @@ -1199,9 +1290,9 @@ test "encode_key concatenates into unambiguous compound keys" { // against the component-wise order they are supposed to reproduce. const gpa = testing.allocator; const parts = [_]Value{ - .{ .string = "" }, .{ .string = "a" }, .{ .string = "a\x00" }, - .{ .string = "ab" }, .{ .int32 = 1 }, .{ .int32 = 2 }, - .null, .{ .array = &.{} }, .{ .bool = true }, + .{ .string = "" }, .{ .string = "a" }, .{ .string = "a\x00" }, + .{ .string = "ab" }, .{ .int32 = 1 }, .{ .int32 = 2 }, + .null, .{ .array = &.{} }, .{ .bool = true }, }; for (parts) |a1| { diff --git a/src/db.zig b/src/db.zig index 74f9f37..e211bde 100644 --- a/src/db.zig +++ b/src/db.zig @@ -320,7 +320,13 @@ pub const Engine = struct { /// shared); the collection lock is acquired before the exclusive catalog /// lock is dropped, so a concurrent drop can never free it underneath. /// Returns null when the collection does not exist (and create is off). - pub fn lock_collection(self: *Engine, db_name: []const u8, coll_name: []const u8, write: bool, create: bool) !?*Collection { + pub fn lock_collection( + self: *Engine, + db_name: []const u8, + coll_name: []const u8, + write: bool, + create: bool, + ) !?*Collection { var coll = self.get_collection(db_name, coll_name); if (coll == null and create) { self.catalog_lock.unlockShared(self.io); @@ -425,7 +431,13 @@ pub const Engine = struct { /// Log an append (and its seq increment) under the log lock, marking /// the append as in flight so a commit leader's seal covers it. - fn log_append(self: *Engine, comptime kind: LogKind, db: []const u8, coll: []const u8, doc: []const u8) !void { + fn log_append( + self: *Engine, + comptime kind: LogKind, + db: []const u8, + coll: []const u8, + doc: []const u8, + ) !void { _ = self.pending_appends.fetchAdd(1, .acq_rel); defer { // The increment above pairs with this decrement on every return @@ -468,12 +480,24 @@ pub const Engine = struct { /// Insert a document. Fails with error.DuplicateKey if the _id exists. /// Generates an ObjectId _id when absent. - pub fn insert(self: *Engine, db_name: []const u8, coll_name: []const u8, doc: *const bson.Document, oid_gen: *bson.ObjectIdGen) !void { + pub fn insert( + self: *Engine, + db_name: []const u8, + coll_name: []const u8, + doc: *const bson.Document, + oid_gen: *bson.ObjectIdGen, + ) !void { return self.upsert(db_name, coll_name, doc, oid_gen, .insert); } /// Insert or replace a document by _id (upsert without existence check). - pub fn replace(self: *Engine, db_name: []const u8, coll_name: []const u8, doc: *const bson.Document, oid_gen: *bson.ObjectIdGen) !void { + pub fn replace( + self: *Engine, + db_name: []const u8, + coll_name: []const u8, + doc: *const bson.Document, + oid_gen: *bson.ObjectIdGen, + ) !void { return self.upsert(db_name, coll_name, doc, oid_gen, .replace); } @@ -574,7 +598,12 @@ pub const Engine = struct { /// Remove a document by its `_id` value. Returns true if it existed. /// The serialized-key encoding stays private to the engine. - pub fn remove_by_id(self: *Engine, db_name: []const u8, coll_name: []const u8, id: bson.Value) !bool { + pub fn remove_by_id( + self: *Engine, + db_name: []const u8, + coll_name: []const u8, + id: bson.Value, + ) !bool { const id_key = try bson.serialize_value(self.gpa, id); defer self.gpa.free(id_key); return self.remove(db_name, coll_name, id_key); @@ -610,7 +639,12 @@ pub const Engine = struct { return db.collections.get(coll_name); } - pub fn get_doc(self: *Engine, db_name: []const u8, coll_name: []const u8, id_key: []const u8) ?[]const u8 { + pub fn get_doc( + self: *Engine, + db_name: []const u8, + coll_name: []const u8, + id_key: []const u8, + ) ?[]const u8 { const coll = self.get_collection(db_name, coll_name) orelse return null; const off = coll.docs.get(id_key) orelse return null; return coll.doc_bytes(off); @@ -636,7 +670,12 @@ pub const Engine = struct { /// after the index builds over the existing documents and passes /// uniqueness, so a rejected create persists nothing. Returns the new /// index (or the existing one when the spec matches — idempotent). - pub fn create_index(self: *Engine, db_name: []const u8, coll_name: []const u8, spec_doc: *const bson.Document) !*index.Index { + pub fn create_index( + self: *Engine, + db_name: []const u8, + coll_name: []const u8, + spec_doc: *const bson.Document, + ) !*index.Index { const coll = try self.get_or_create_collection(db_name, coll_name); var ix = try index.parse_spec(self.gpa, spec_doc); var committed = false; @@ -675,7 +714,12 @@ pub const Engine = struct { /// Remove a secondary index by name, persisting a drop record first. /// Returns false when no such index exists. - pub fn drop_index(self: *Engine, db_name: []const u8, coll_name: []const u8, index_name: []const u8) !bool { + pub fn drop_index( + self: *Engine, + db_name: []const u8, + coll_name: []const u8, + index_name: []const u8, + ) !bool { const db = self.dbs.get(db_name) orelse return false; const coll = db.collections.get(coll_name) orelse return false; if (coll.find_index(index_name) == null) return false; @@ -722,7 +766,13 @@ pub const Engine = struct { /// Sweep one collection under its write lock; the lock is released on /// every return path. Returns how many documents were removed. - fn ttl_sweep_coll(self: *Engine, coll: *Collection, now_ms: i64, db_name: []const u8, coll_name: []const u8) !usize { + fn ttl_sweep_coll( + self: *Engine, + coll: *Collection, + now_ms: i64, + db_name: []const u8, + coll_name: []const u8, + ) !usize { try coll.lock.lock(self.io); defer coll.lock.unlock(self.io); // Ids are duped rather than aliased: `remove` frees the docs-map key @@ -778,7 +828,11 @@ pub const Engine = struct { while (it.next()) |entry| try out.append(self.gpa, entry.key_ptr.*); } - pub fn collection_names(self: *Engine, db_name: []const u8, out: *std.ArrayListUnmanaged([]const u8)) !void { + pub fn collection_names( + self: *Engine, + db_name: []const u8, + out: *std.ArrayListUnmanaged([]const u8), + ) !void { const db = self.dbs.get(db_name) orelse return; var it = db.collections.iterator(); while (it.next()) |entry| try out.append(self.gpa, entry.key_ptr.*); @@ -786,7 +840,11 @@ pub const Engine = struct { // -- internals ----------------------------------------------------------- - pub fn get_or_create_collection(self: *Engine, db_name: []const u8, coll_name: []const u8) !*Collection { + pub fn get_or_create_collection( + self: *Engine, + db_name: []const u8, + coll_name: []const u8, + ) !*Collection { const db = self.dbs.getPtr(db_name) orelse { const db_key = try self.gpa.dupe(u8, db_name); errdefer self.gpa.free(db_key); @@ -808,7 +866,11 @@ pub const Engine = struct { /// generated ObjectId `_id` when absent. /// The canonical bytes of `doc`, with an ObjectId `_id` generated when /// absent. The result is owned by the caller. - fn serialize_with_id(self: *Engine, doc: *const bson.Document, oid_gen: *bson.ObjectIdGen) ![]u8 { + fn serialize_with_id( + self: *Engine, + doc: *const bson.Document, + oid_gen: *bson.ObjectIdGen, + ) ![]u8 { if (doc.get("_id") != null) return serialize_doc(self.gpa, doc); var pairs: std.ArrayListUnmanaged(bson.Pair) = .empty; defer pairs.deinit(self.gpa); @@ -994,7 +1056,13 @@ pub const Engine = struct { /// Re-emit one collection's index specs and documents into the compacted /// log, under the collection's write lock (released on every return /// path, including errors). - fn compact_snapshot_coll(self: *Engine, coll: *Collection, new_log: *storage.Log, db_name: []const u8, coll_name: []const u8) !void { + fn compact_snapshot_coll( + self: *Engine, + coll: *Collection, + new_log: *storage.Log, + db_name: []const u8, + coll_name: []const u8, + ) !void { try coll.lock.lock(self.io); defer coll.lock.unlock(self.io); // Re-emit the index definitions first: a compacted log that dropped @@ -1045,7 +1113,13 @@ pub const Engine = struct { while (doc_it.next()) |doc_entry| { ix.append_doc_entries(self.gpa, coll.doc_bytes(doc_entry.value_ptr.*), doc_entry.key_ptr.*) catch |err| switch (err) { error.ParallelArrays => { - std.debug.print("multiforadb: WARNING: index '{s}' cannot index an existing document; entry skipped\n", .{ix.name}); + std.debug.print( + "multiforadb: WARNING: index '{s}' cannot index an existing " ++ + "document; entry skipped\n", + .{ + ix.name, + }, + ); continue; }, else => return err, @@ -1053,13 +1127,23 @@ pub const Engine = struct { } // Tolerated, not enforced: the database must always open. if (try ix.finish_bulk(self.gpa, false)) { - std.debug.print("multiforadb: WARNING: unique index '{s}' has duplicate keys in existing data; duplicates not enforced for existing documents\n", .{ix.name}); + std.debug.print( + "multiforadb: WARNING: unique index '{s}' has duplicate keys in existing " ++ + "data; duplicates not enforced for existing documents\n", + .{ + ix.name, + }, + ); } } /// Register an (empty) index from a persisted spec document. A repeated /// create record for the same name is an idempotent no-op. - fn register_index_from_spec(self: *Engine, coll: *Collection, spec_doc: *const bson.Document) !void { + fn register_index_from_spec( + self: *Engine, + coll: *Collection, + spec_doc: *const bson.Document, + ) !void { var ix = try index.parse_spec(self.gpa, spec_doc); var committed = false; defer if (!committed) ix.deinit(self.gpa); @@ -1103,7 +1187,9 @@ fn apply_record(ctx: *anyopaque, record: storage.Record, doc: *bson.Document) an switch (record.type) { storage.record_type_index_create => { self.register_index_from_spec(coll, doc) catch |err| { - std.debug.print("multiforadb: index create record failed to apply: {s}\n", .{@errorName(err)}); + std.debug.print("multiforadb: index create record failed to apply: {s}\n", .{ + @errorName(err), + }); return; }; return; @@ -1455,7 +1541,12 @@ test "concurrent readers and writers on a threaded Io" { var remaining = std.atomic.Value(usize).init(@intCast(total)); const Worker = struct { - fn writer(e: *Engine, id_counter: *std.atomic.Value(i32), pending: *std.atomic.Value(usize), alloc: std.mem.Allocator) error{Canceled}!void { + fn writer( + e: *Engine, + id_counter: *std.atomic.Value(i32), + pending: *std.atomic.Value(usize), + alloc: std.mem.Allocator, + ) error{Canceled}!void { while (true) { const id = id_counter.fetchAdd(1, .monotonic); if (id > total) return; @@ -1605,7 +1696,11 @@ test "concurrent writers compacting: the log survives a reopen" { } } - fn writer(e: *Engine, id_counter: *std.atomic.Value(i32), alloc: std.mem.Allocator) error{Canceled}!void { + fn writer( + e: *Engine, + id_counter: *std.atomic.Value(i32), + alloc: std.mem.Allocator, + ) error{Canceled}!void { while (true) { const id = id_counter.fetchAdd(1, .monotonic); if (id > total) return; @@ -1645,7 +1740,14 @@ test "concurrent writers compacting: the log survives a reopen" { /// A spec document for a single-path index, built by serializing and /// re-parsing so the pairs are arena-owned. -fn index_spec(gpa: std.mem.Allocator, path: []const u8, name: []const u8, unique: bool, sparse: bool, ttl: ?i64) !bson.Document { +fn index_spec( + gpa: std.mem.Allocator, + path: []const u8, + name: []const u8, + unique: bool, + sparse: bool, + ttl: ?i64, +) !bson.Document { var out: std.ArrayListUnmanaged(u8) = .empty; defer out.deinit(gpa); var pairs: std.ArrayListUnmanaged(bson.Pair) = .empty; @@ -1662,7 +1764,14 @@ fn index_spec(gpa: std.mem.Allocator, path: []const u8, name: []const u8, unique } /// Number of entries the named index has for a single-value equality key. -fn index_count(gpa: std.mem.Allocator, engine: *Engine, db_name: []const u8, coll_name: []const u8, name: []const u8, key_value: bson.Value) !usize { +fn index_count( + gpa: std.mem.Allocator, + engine: *Engine, + db_name: []const u8, + coll_name: []const u8, + name: []const u8, + key_value: bson.Value, +) !usize { const coll = engine.get_collection(db_name, coll_name) orelse return 0; for (coll.indexes.items) |*ix| { if (std.mem.eql(u8, ix.name, name)) { diff --git a/src/fuzz_split.zig b/src/fuzz_split.zig index 9c36ae6..f626521 100644 --- a/src/fuzz_split.zig +++ b/src/fuzz_split.zig @@ -113,7 +113,12 @@ fn run(seed: u64, ops: usize, max_len: usize) !void { if (std.mem.eql(u8, got, d.id)) hit = true; } if (!hit) { - std.debug.print("seed {d} op {d}: id {s} (key len {d}) not found by descent\n", .{ seed, op, d.id, d.s.len }); + std.debug.print("seed {d} op {d}: id {s} (key len {d}) not found by descent\n", .{ + seed, + op, + d.id, + d.s.len, + }); return error.EntryUnreachable; } } diff --git a/src/main.zig b/src/main.zig index 8d57ebb..fe57d94 100644 --- a/src/main.zig +++ b/src/main.zig @@ -99,7 +99,10 @@ pub fn main(init: std.process.Init) !void { var engine = try mongo.db.Engine.open(init.gpa, init.io, db_path); defer engine.deinit(); engine.compact_threshold = compact_threshold; - std.debug.print("multiforadb: opened database '{s}' (compact threshold {d})\n", .{ db_path, compact_threshold }); + std.debug.print("multiforadb: opened database '{s}' (compact threshold {d})\n", .{ + db_path, + compact_threshold, + }); var server = mongo.server.Server{ .gpa = init.gpa, diff --git a/src/query.zig b/src/query.zig index 0334729..18f799b 100644 --- a/src/query.zig +++ b/src/query.zig @@ -14,7 +14,11 @@ const bson = @import("bson.zig"); /// input — but it must still be a possible error, not a panic. pub const QueryError = error{ OutOfMemory, InvalidBson }; -pub fn matches(gpa: std.mem.Allocator, filter: *const bson.Document, doc: *const bson.Document) QueryError!bool { +pub fn matches( + gpa: std.mem.Allocator, + filter: *const bson.Document, + doc: *const bson.Document, +) QueryError!bool { for (filter.pairs) |p| { if (p.key.len > 0 and p.key[0] == '$') { if (!try match_top_level(gpa, p.key, p.value, doc)) return false; @@ -25,7 +29,12 @@ pub fn matches(gpa: std.mem.Allocator, filter: *const bson.Document, doc: *const return true; } -fn match_top_level(gpa: std.mem.Allocator, op: []const u8, value: bson.Value, doc: *const bson.Document) QueryError!bool { +fn match_top_level( + gpa: std.mem.Allocator, + op: []const u8, + value: bson.Value, + doc: *const bson.Document, +) QueryError!bool { if (std.mem.eql(u8, op, "$and") or std.mem.eql(u8, op, "$or")) { const want_and = std.mem.eql(u8, op, "$and"); const filters = switch (value) { @@ -89,7 +98,12 @@ fn is_operator_doc(value: bson.Value) ?[]const bson.Pair { /// collection scan. const inline_candidates = 8; -fn field_matches(gpa: std.mem.Allocator, path: []const u8, expected: bson.Value, doc: *const bson.Document) QueryError!bool { +fn field_matches( + gpa: std.mem.Allocator, + path: []const u8, + expected: bson.Value, + doc: *const bson.Document, +) QueryError!bool { var stack_fallback = std.heap.stackFallback(inline_candidates * @sizeOf(bson.Value), gpa); const alloc = stack_fallback.get(); @@ -103,7 +117,12 @@ fn field_matches(gpa: std.mem.Allocator, path: []const u8, expected: bson.Value, /// The byte counterpart of field_matches: collects values by walking the /// canonical BSON element stream of a stored document, skipping by length /// any field the filter does not name. -fn field_matches_bytes(gpa: std.mem.Allocator, path: []const u8, expected: bson.Value, bytes: []const u8) QueryError!bool { +fn field_matches_bytes( + gpa: std.mem.Allocator, + path: []const u8, + expected: bson.Value, + bytes: []const u8, +) QueryError!bool { // An arena, not a stack fallback: the byte walker materializes nested // doc/array values (whole-array equality, embedded docs) into the // allocator it is given, and those must be freed with it. @@ -120,7 +139,10 @@ fn field_matches_bytes(gpa: std.mem.Allocator, path: []const u8, expected: bson. /// MongoDB applies queries to array elements as well as the array itself. /// Index the snapshot length, re-reading items each iteration: appending /// may reallocate the buffer, which would invalidate a captured slice. -fn expand_arrays(alloc: std.mem.Allocator, candidates: *std.ArrayListUnmanaged(bson.Value)) QueryError!void { +fn expand_arrays( + alloc: std.mem.Allocator, + candidates: *std.ArrayListUnmanaged(bson.Value), +) QueryError!void { const direct_count = candidates.items.len; var i: usize = 0; while (i < direct_count) : (i += 1) { @@ -133,7 +155,11 @@ fn expand_arrays(alloc: std.mem.Allocator, candidates: *std.ArrayListUnmanaged(b /// The operator/equality half of field matching, shared by the tree and /// byte collectors. -fn apply_expected(gpa: std.mem.Allocator, expected: bson.Value, candidates: []const bson.Value) QueryError!bool { +fn apply_expected( + gpa: std.mem.Allocator, + expected: bson.Value, + candidates: []const bson.Value, +) QueryError!bool { if (is_operator_doc(expected)) |pairs| { // $options modifies $regex wherever it appears in the document, so // it has to be known before any operator runs. @@ -166,7 +192,11 @@ fn apply_expected(gpa: std.mem.Allocator, expected: bson.Value, candidates: []co /// byte-matcher counterpart of `matches`, used by scans. Same semantics, /// different collection: fields the filter does not name are skipped by /// length instead of materialized. -pub fn matches_bytes(gpa: std.mem.Allocator, filter: []const bson.Pair, bytes: []const u8) QueryError!bool { +pub fn matches_bytes( + gpa: std.mem.Allocator, + filter: []const bson.Pair, + bytes: []const u8, +) QueryError!bool { for (filter) |p| { if (p.key.len > 0 and p.key[0] == '$') { if (!try match_top_level_bytes(gpa, p.key, p.value, bytes)) return false; @@ -177,7 +207,12 @@ pub fn matches_bytes(gpa: std.mem.Allocator, filter: []const bson.Pair, bytes: [ return true; } -fn match_top_level_bytes(gpa: std.mem.Allocator, op: []const u8, value: bson.Value, bytes: []const u8) QueryError!bool { +fn match_top_level_bytes( + gpa: std.mem.Allocator, + op: []const u8, + value: bson.Value, + bytes: []const u8, +) QueryError!bool { if (std.mem.eql(u8, op, "$and") or std.mem.eql(u8, op, "$or")) { const want_and = std.mem.eql(u8, op, "$and"); const filters = switch (value) { @@ -215,7 +250,13 @@ fn match_top_level_bytes(gpa: std.mem.Allocator, op: []const u8, value: bson.Val /// Collect values reachable at `path` from a document's canonical bytes — /// the byte counterpart of `collect_values`, with the same traversal, the /// same order and the same multikey semantics. Appends into `out`. -pub fn collect_values_bytes(gpa: std.mem.Allocator, bytes: []const u8, path: []const u8, out: *std.ArrayListUnmanaged(bson.Value), depth: usize) QueryError!void { +pub fn collect_values_bytes( + gpa: std.mem.Allocator, + bytes: []const u8, + path: []const u8, + out: *std.ArrayListUnmanaged(bson.Value), + depth: usize, +) QueryError!void { var it = std.mem.splitScalar(u8, path, '.'); const first = it.next() orelse return; const rest = it.rest(); @@ -241,7 +282,15 @@ pub fn collect_values_bytes(gpa: std.mem.Allocator, bytes: []const u8, path: []c } } -fn collect_from_value_bytes(gpa: std.mem.Allocator, bytes: []const u8, idx: *usize, tag: u8, path: []const u8, out: *std.ArrayListUnmanaged(bson.Value), depth: usize) QueryError!void { +fn collect_from_value_bytes( + gpa: std.mem.Allocator, + bytes: []const u8, + idx: *usize, + tag: u8, + path: []const u8, + out: *std.ArrayListUnmanaged(bson.Value), + depth: usize, +) QueryError!void { if (depth > 8) { try bson.skip_value(bytes, idx, tag); return; @@ -345,7 +394,13 @@ fn parse_op(name: []const u8) Op { return op_names.get(name) orelse .unknown; } -fn match_operator(gpa: std.mem.Allocator, op: Op, value: bson.Value, actuals: []const bson.Value, regex_options: []const u8) QueryError!bool { +fn match_operator( + gpa: std.mem.Allocator, + op: Op, + value: bson.Value, + actuals: []const bson.Value, + regex_options: []const u8, +) QueryError!bool { if (op == .eq) { for (actuals) |a| if (bson.compare(a, value) == .eq) return true; return false; @@ -489,7 +544,13 @@ fn match_operator(gpa: std.mem.Allocator, op: Op, value: bson.Value, actuals: [] /// Appends into `out`; on OOM, collection stops early (the engine is /// already failing at that point). Public because index entry generation /// must mirror field_matches exactly (src/index.zig). -pub fn collect_values(gpa: std.mem.Allocator, pairs: []const bson.Pair, path: []const u8, out: *std.ArrayListUnmanaged(bson.Value), depth: usize) QueryError!void { +pub fn collect_values( + gpa: std.mem.Allocator, + pairs: []const bson.Pair, + path: []const u8, + out: *std.ArrayListUnmanaged(bson.Value), + depth: usize, +) QueryError!void { var it = std.mem.splitScalar(u8, path, '.'); const first = it.next() orelse return; @@ -506,7 +567,13 @@ pub fn collect_values(gpa: std.mem.Allocator, pairs: []const bson.Pair, path: [] } } -fn collect_from_value(gpa: std.mem.Allocator, v: bson.Value, path: []const u8, out: *std.ArrayListUnmanaged(bson.Value), depth: usize) QueryError!void { +fn collect_from_value( + gpa: std.mem.Allocator, + v: bson.Value, + path: []const u8, + out: *std.ArrayListUnmanaged(bson.Value), + depth: usize, +) QueryError!void { if (depth > 8) return; switch (v) { .doc => |pairs| try collect_values(gpa, pairs, path, out, depth), @@ -581,7 +648,14 @@ pub fn regex_match(pattern: []const u8, options: []const u8, text: []const u8) b /// Match `pattern[p..]` against `text[t..]`, returning the new text /// position on success (null on failure). Backtracks via recursion. -fn match_here(pattern: []const u8, p: *usize, text: []const u8, t: usize, ci: bool, dot_all: bool) ?usize { +fn match_here( + pattern: []const u8, + p: *usize, + text: []const u8, + t: usize, + ci: bool, + dot_all: bool, +) ?usize { var pos = t; while (p.* < pattern.len) { const c = pattern[p.*]; @@ -669,7 +743,9 @@ fn match_here(pattern: []const u8, p: *usize, text: []const u8, t: usize, ci: bo var q_end = element_end; var min: usize = 1; var max: usize = 1; - if (element_end < pattern.len and (pattern[element_end] == '*' or pattern[element_end] == '+' or pattern[element_end] == '?')) { + if (element_end < pattern.len and (pattern[element_end] == '*' or pattern[element_end] == '+' or pattern[ + element_end + ] == '?')) { switch (pattern[element_end]) { '*' => { min = 0; @@ -847,7 +923,11 @@ const SortCtx = struct { /// Pull each document's sort-key values into one flat allocation, so the /// comparator is pure and cannot fail. -fn decorate(arena: std.mem.Allocator, docs: []*const bson.Document, keys: []const SortKey) QueryError![]SortedDoc { +fn decorate( + arena: std.mem.Allocator, + docs: []*const bson.Document, + keys: []const SortKey, +) QueryError![]SortedDoc { const entries = try arena.alloc(SortedDoc, docs.len); const flat = try arena.alloc(bson.Value, docs.len * keys.len); for (docs, 0..) |d, i| { @@ -861,7 +941,11 @@ fn decorate(arena: std.mem.Allocator, docs: []*const bson.Document, keys: []cons } /// Sort `docs` in place by `keys`. -pub fn sort_docs(arena: std.mem.Allocator, docs: []*const bson.Document, keys: []const SortKey) QueryError!void { +pub fn sort_docs( + arena: std.mem.Allocator, + docs: []*const bson.Document, + keys: []const SortKey, +) QueryError!void { if (keys.len == 0 or docs.len < 2) return; const entries = try decorate(arena, docs, keys); std.mem.sort(SortedDoc, entries, SortCtx{ .keys = keys }, SortCtx.less); @@ -876,7 +960,12 @@ pub fn sort_docs(arena: std.mem.Allocator, docs: []*const bson.Document, keys: [ /// n log n comparisons to discard almost all of the result. This keeps a /// k-element max-heap instead: one comparison against the heap root per /// document, and only the survivors are ever ordered. -pub fn sort_docs_top_k(arena: std.mem.Allocator, docs: []*const bson.Document, keys: []const SortKey, k: usize) QueryError!void { +pub fn sort_docs_top_k( + arena: std.mem.Allocator, + docs: []*const bson.Document, + keys: []const SortKey, + k: usize, +) QueryError!void { if (keys.len == 0 or docs.len < 2) return; if (k == 0) return; if (k >= docs.len) return sort_docs(arena, docs, keys); @@ -936,7 +1025,12 @@ pub const ProjectionError = std.mem.Allocator.Error; /// Apply a projection document, writing resulting pairs into `out` (which /// should use the caller's arena so strings are owned). -pub fn project(arena: std.mem.Allocator, doc: *const bson.Document, proj: *const bson.Document, out: *std.ArrayListUnmanaged(bson.Pair)) ProjectionError!void { +pub fn project( + arena: std.mem.Allocator, + doc: *const bson.Document, + proj: *const bson.Document, + out: *std.ArrayListUnmanaged(bson.Pair), +) ProjectionError!void { var inclusion: ?bool = null; var non_id_count: usize = 0; for (proj.pairs) |p| { @@ -984,7 +1078,12 @@ pub fn project(arena: std.mem.Allocator, doc: *const bson.Document, proj: *const } /// Recursively apply exclusions to a nested document given the parent path. -fn exclude_doc(arena: std.mem.Allocator, pairs: []const bson.Pair, proj: *const bson.Document, parent: []const u8) ProjectionError![]const bson.Pair { +fn exclude_doc( + arena: std.mem.Allocator, + pairs: []const bson.Pair, + proj: *const bson.Document, + parent: []const u8, +) ProjectionError![]const bson.Pair { var out: std.ArrayListUnmanaged(bson.Pair) = .empty; errdefer out.deinit(arena); for (pairs) |p| { @@ -1024,7 +1123,12 @@ pub fn truthy(v: bson.Value) bool { } /// Include a dotted path (e.g. "a.b.c"), creating nested documents as needed. -fn project_path(arena: std.mem.Allocator, pairs: []const bson.Pair, path: []const u8, out: *std.ArrayListUnmanaged(bson.Pair)) ProjectionError!void { +fn project_path( + arena: std.mem.Allocator, + pairs: []const bson.Pair, + path: []const u8, + out: *std.ArrayListUnmanaged(bson.Pair), +) ProjectionError!void { var it = std.mem.splitScalar(u8, path, '.'); const first = it.next() orelse return; const rest = it.rest(); @@ -1294,10 +1398,10 @@ test "first_value_at agrees with collect_values on its first element" { }); const paths = [_][]const u8{ - "n", "sub", "sub.x", "sub.y", "sub.missing", - "items", "items.v", "items.0", "items.1.v", "items.9", - "nums", "nums.0", "nums.1", "nums.5", - "dup", "missing", "n.deeper", "", "sub.x.y", + "n", "sub", "sub.x", "sub.y", "sub.missing", + "items", "items.v", "items.0", "items.1.v", "items.9", + "nums", "nums.0", "nums.1", "nums.5", "dup", + "missing", "n.deeper", "", "sub.x.y", }; for (paths) |path| { @@ -1308,12 +1412,17 @@ test "first_value_at agrees with collect_values on its first element" { const first = first_value_at(d.pairs, path, 0); if (list.items.len == 0) { testing.expect(first == null) catch |e| { - std.debug.print("path '{s}': collect empty but first_value_at returned a value\n", .{path}); + std.debug.print("path '{s}': collect empty but first_value_at returned a value\n", .{ + path, + }); return e; }; } else { testing.expect(first != null) catch |e| { - std.debug.print("path '{s}': collect got {d} values but first_value_at returned null\n", .{ path, list.items.len }); + std.debug.print("path '{s}': collect got {d} values but first_value_at returned null\n", .{ + path, + list.items.len, + }); return e; }; testing.expectEqual(std.math.Order.eq, bson.compare(list.items[0], first.?)) catch |e| { @@ -1372,7 +1481,11 @@ test "top-k selection matches a full sort on the leading page" { const want = first_value_at(full[i].pairs, sk.path, 0) orelse bson.Value.null; const got = first_value_at(topk[i].pairs, sk.path, 0) orelse bson.Value.null; testing.expectEqual(std.math.Order.eq, bson.compare(want, got)) catch |e| { - std.debug.print("k={d} pos={d} key='{s}' diverged from the full sort\n", .{ k, i, sk.path }); + std.debug.print("k={d} pos={d} key='{s}' diverged from the full sort\n", .{ + k, + i, + sk.path, + }); return e; }; } @@ -1482,7 +1595,7 @@ test "byte matcher agrees with the tree matcher on a corpus" { np += 1; } if (rand.boolean()) { - pairs[np] = .{ .key = "d", .value = .{ .doc = &.{ .{ .key = "e", .value = .{ .int32 = a } } } } }; + pairs[np] = .{ .key = "d", .value = .{ .doc = &.{.{ .key = "e", .value = .{ .int32 = a } }} } }; np += 1; } var out: std.ArrayListUnmanaged(u8) = .empty; @@ -1541,7 +1654,11 @@ test "byte matcher agrees with the tree matcher on a corpus" { const tree = try matches(gpa, &filter_doc, &doc); const byt = try matches_bytes(gpa, f_pairs.items, bytes); if (tree != byt) { - std.debug.print("case {d}: filter mismatch: tree={} bytes={}\n", .{ case, tree, byt }); + std.debug.print("case {d}: filter mismatch: tree={} bytes={}\n", .{ + case, + tree, + byt, + }); return error.ByteMatcherMismatch; } } @@ -1549,7 +1666,12 @@ test "byte matcher agrees with the tree matcher on a corpus" { } /// Public single-value operator matcher, used by $pull and $elemMatch. -pub fn value_matches_operator(gpa: std.mem.Allocator, op: []const u8, value: bson.Value, actual: bson.Value) QueryError!bool { +pub fn value_matches_operator( + gpa: std.mem.Allocator, + op: []const u8, + value: bson.Value, + actual: bson.Value, +) QueryError!bool { var single: [1]bson.Value = .{actual}; return match_operator(gpa, parse_op(op), value, single[0..], ""); } diff --git a/src/server.zig b/src/server.zig index 9ee0ef6..d94fcc1 100644 --- a/src/server.zig +++ b/src/server.zig @@ -120,7 +120,10 @@ fn handle_connection_inner(io: std.Io, stream: std.Io.net.Stream, server: *Serve reader.interface.readSliceAll(&len_bytes) catch return; // clean client disconnect (EOF or RST) const total: u32 = std.mem.readInt(u32, &len_bytes, .little); if (total < 16 or total > wire.max_message_size) { - std.debug.print("multiforadb: bad message length {d} on conn {d}\n", .{ total, connection_id }); + std.debug.print("multiforadb: bad message length {d} on conn {d}\n", .{ + total, + connection_id, + }); return; } @@ -129,14 +132,22 @@ fn handle_connection_inner(io: std.Io, stream: std.Io.net.Stream, server: *Serve msg_buf.items.len = total; std.mem.writeInt(u32, msg_buf.items[0..4], total, .little); reader.interface.readSliceAll(msg_buf.items[4..]) catch |err| { - std.debug.print("multiforadb: read error on conn {d}: {s} (body, len {d})\n", .{ connection_id, @errorName(err), total }); + std.debug.print("multiforadb: read error on conn {d}: {s} (body, len {d})\n", .{ + connection_id, + @errorName(err), + total, + }); return; }; var msg = wire.Message.parse(server.gpa, msg_buf.items) catch |err| { // Unparseable request: close the connection. const op: i32 = if (msg_buf.items.len >= 16) std.mem.readInt(i32, msg_buf.items[12..16], .little) else 0; - std.debug.print("multiforadb: bad message on conn {d}: {s} (opCode {d})\n", .{ connection_id, @errorName(err), op }); + std.debug.print("multiforadb: bad message on conn {d}: {s} (opCode {d})\n", .{ + connection_id, + @errorName(err), + op, + }); return; }; defer msg.deinit(); @@ -146,7 +157,11 @@ fn handle_connection_inner(io: std.Io, stream: std.Io.net.Stream, server: *Serve commands.dispatch(&ctx, &msg, &reply) catch |err| { // Discard any partial reply (the client would read the first // ok field, which may already say 1) and send a clean error. - std.debug.print("multiforadb: dispatch error on conn {d} cmd {s}: {s}\n", .{ connection_id, msg.command_name(), @errorName(err) }); + std.debug.print("multiforadb: dispatch error on conn {d} cmd {s}: {s}\n", .{ + connection_id, + msg.command_name(), + @errorName(err), + }); reply.pairs.clearRetainingCapacity(); reply.put_error( @intFromEnum(commands.ErrorCode.internal_error), @@ -161,16 +176,25 @@ fn handle_connection_inner(io: std.Io, stream: std.Io.net.Stream, server: *Serve else reply.build(server.gpa, reply_request_id, msg.request_id, &out_buf); built catch |err| { - std.debug.print("multiforadb: reply build error on conn {d}: {s}\n", .{ connection_id, @errorName(err) }); + std.debug.print("multiforadb: reply build error on conn {d}: {s}\n", .{ + connection_id, + @errorName(err), + }); return; }; reply_request_id +%= 1; writer.interface.writeAll(out_buf.items) catch |err| { - std.debug.print("multiforadb: write error on conn {d}: {s}\n", .{ connection_id, @errorName(err) }); + std.debug.print("multiforadb: write error on conn {d}: {s}\n", .{ + connection_id, + @errorName(err), + }); return; }; writer.interface.flush() catch |err| { - std.debug.print("multiforadb: flush error on conn {d}: {s}\n", .{ connection_id, @errorName(err) }); + std.debug.print("multiforadb: flush error on conn {d}: {s}\n", .{ + connection_id, + @errorName(err), + }); return; }; } diff --git a/src/spill.zig b/src/spill.zig index 1bca2c5..cf44ba5 100644 --- a/src/spill.zig +++ b/src/spill.zig @@ -37,7 +37,12 @@ pub fn main() !void { docs[i] = try doc_of(gpa, &pairs); _ = try ix.add_doc(gpa, docs[i], &[_]u8{ 'i', 'd', @intCast(i + 1) }, true); } - std.debug.print("count={d} leaves={d} depth={d} overflow={d}\n", .{ ix.count(), ix.leaf_count, ix.depth, ix.overflow.items.len }); + std.debug.print("count={d} leaves={d} depth={d} overflow={d}\n", .{ + ix.count(), + ix.leaf_count, + ix.depth, + ix.overflow.items.len, + }); if (ix.overflow.items.len < 100_000) return error.NoSpill; // Every entry is found by exact key. diff --git a/src/spill2.zig b/src/spill2.zig index 3af7c9f..4820dc5 100644 --- a/src/spill2.zig +++ b/src/spill2.zig @@ -48,7 +48,12 @@ pub fn main() !void { const d = try doc_of(gpa, &pairs); _ = try ix.add_doc(gpa, d, id, false); } - std.debug.print("count={d} leaves={d} depth={d} overflow={d}\n", .{ ix.count(), ix.leaf_count, ix.depth, ix.overflow.items.len }); + std.debug.print("count={d} leaves={d} depth={d} overflow={d}\n", .{ + ix.count(), + ix.leaf_count, + ix.depth, + ix.overflow.items.len, + }); if (ix.count() != N) return error.Bad; // Spot-check exact lookups. diff --git a/src/storage.zig b/src/storage.zig index cf8ee1b..e3330dd 100644 --- a/src/storage.zig +++ b/src/storage.zig @@ -227,7 +227,10 @@ pub const Log = struct { while (true) { var hdr: [block_header_len]u8 = undefined; const n = self.file.readPositionalAll(self.io, &hdr, pos) catch |err| { - std.debug.print("multiforadb: log read error at {d}: {s}\n", .{ pos, @errorName(err) }); + std.debug.print("multiforadb: log read error at {d}: {s}\n", .{ + pos, + @errorName(err), + }); return error.InvalidLog; }; if (n == 0) return; // clean end @@ -262,7 +265,10 @@ pub const Log = struct { codec_raw => try decomp.appendSlice(self.gpa, payload), codec_lz4 => try lz4_decompress(self.gpa, payload, &decomp), else => { - std.debug.print("multiforadb: unknown block codec {d} at {d}\n", .{ codec, pos }); + std.debug.print("multiforadb: unknown block codec {d} at {d}\n", .{ + codec, + pos, + }); return error.InvalidLog; }, } @@ -280,7 +286,13 @@ pub const Log = struct { /// and deliver it. Returns the record's byte length. Any framing failure /// here is interior corruption: a block was sealed only with complete /// records, and its hash proved the stored bytes intact. - fn parse_record(self: *Log, bytes: []const u8, pos: u64, ctx: *anyopaque, callback: ReplayFn) !usize { + fn parse_record( + self: *Log, + bytes: []const u8, + pos: u64, + ctx: *anyopaque, + callback: ReplayFn, + ) !usize { if (bytes.len < 4) return error.InvalidLog; const total: u32 = std.mem.readInt(u32, bytes[0..4], .little); if (total < header_len) { @@ -321,26 +333,57 @@ pub const Log = struct { return total; } - pub fn append_upsert(self: *Log, db: []const u8, coll: []const u8, doc: []const u8, seq: u64) !void { + pub fn append_upsert( + self: *Log, + db: []const u8, + coll: []const u8, + doc: []const u8, + seq: u64, + ) !void { try self.append(record_type_upsert, db, coll, doc, seq); } - pub fn append_delete(self: *Log, db: []const u8, coll: []const u8, doc: []const u8, seq: u64) !void { + pub fn append_delete( + self: *Log, + db: []const u8, + coll: []const u8, + doc: []const u8, + seq: u64, + ) !void { try self.append(record_type_delete, db, coll, doc, seq); } /// The payload is the canonical index spec document ({v, key, name, /// unique?, sparse?}); only apply_record interprets it. - pub fn append_index_create(self: *Log, db: []const u8, coll: []const u8, doc: []const u8, seq: u64) !void { + pub fn append_index_create( + self: *Log, + db: []const u8, + coll: []const u8, + doc: []const u8, + seq: u64, + ) !void { try self.append(record_type_index_create, db, coll, doc, seq); } /// The payload is {name: "..."}; only apply_record interprets it. - pub fn append_index_drop(self: *Log, db: []const u8, coll: []const u8, doc: []const u8, seq: u64) !void { + pub fn append_index_drop( + self: *Log, + db: []const u8, + coll: []const u8, + doc: []const u8, + seq: u64, + ) !void { try self.append(record_type_index_drop, db, coll, doc, seq); } - fn append(self: *Log, rtype: u8, db: []const u8, coll: []const u8, doc: []const u8, seq: u64) !void { + fn append( + self: *Log, + rtype: u8, + db: []const u8, + coll: []const u8, + doc: []const u8, + seq: u64, + ) !void { if (std.mem.indexOfScalar(u8, db, 0) != null or std.mem.indexOfScalar(u8, coll, 0) != null) { return error.NulInName; } @@ -547,7 +590,11 @@ fn emit_literals(dst: []u8, literals: []const u8) usize { /// Decompress an LZ4 block into `out` (appended). The block hash has /// already proven the input intact when this runs during replay, so /// structural failures here mean a bug or a raw-codec mismatch. -fn lz4_decompress(gpa: std.mem.Allocator, src: []const u8, out: *std.ArrayListUnmanaged(u8)) error{ CorruptLz4, OutOfMemory }!void { +fn lz4_decompress( + gpa: std.mem.Allocator, + src: []const u8, + out: *std.ArrayListUnmanaged(u8), +) error{ CorruptLz4, OutOfMemory }!void { var ip: usize = 0; while (ip < src.len) { const token = src[ip]; diff --git a/src/stress.zig b/src/stress.zig index e0e5630..01c45b7 100644 --- a/src/stress.zig +++ b/src/stress.zig @@ -16,7 +16,15 @@ fn doc_of(gpa: std.mem.Allocator, pairs: []const bson.Pair) ![]u8 { const Fact = struct { a: i32, b: i32 }; -fn check_range(gpa: std.mem.Allocator, ix: *const index.Index, prefix: bson.Value, lo: ?bson.Value, hi: ?bson.Value, facts: []const Fact, alive: []const bool) !void { +fn check_range( + gpa: std.mem.Allocator, + ix: *const index.Index, + prefix: bson.Value, + lo: ?bson.Value, + hi: ?bson.Value, + facts: []const Fact, + alive: []const bool, +) !void { var out: std.ArrayListUnmanaged([]const u8) = .empty; defer out.deinit(gpa); try ix.lookup_range(gpa, &.{prefix}, lo, true, hi, false, &out); @@ -29,7 +37,13 @@ fn check_range(gpa: std.mem.Allocator, ix: *const index.Index, prefix: bson.Valu expected += 1; } if (out.items.len != expected) { - std.debug.print("MISMATCH: prefix={d} lo={?d} hi={?d}: got {d}, want {d}\n", .{ prefix.int32, if (lo) |l| l.int32 else null, if (hi) |h| h.int32 else null, out.items.len, expected }); + std.debug.print("MISMATCH: prefix={d} lo={?d} hi={?d}: got {d}, want {d}\n", .{ + prefix.int32, + if (lo) |l| l.int32 else null, + if (hi) |h| h.int32 else null, + out.items.len, + expected, + }); std.process.exit(1); } } @@ -61,7 +75,12 @@ pub fn main() !void { try ix.append_doc_entries(gpa, d, id); } _ = try ix.finish_bulk(gpa, false); - std.debug.print("bulk: count={d} leaves={d} depth={d} nodes={d}\n", .{ ix.count(), ix.leaf_count, ix.depth, ix.nodes.items.len }); + std.debug.print("bulk: count={d} leaves={d} depth={d} nodes={d}\n", .{ + ix.count(), + ix.leaf_count, + ix.depth, + ix.nodes.items.len, + }); if (ix.count() != N) return error.BadCount; // Random range checks against brute force. @@ -88,7 +107,12 @@ pub fn main() !void { const d = try doc_of(gpa, &pairs); _ = try ix.add_doc(gpa, d, id, false); } - std.debug.print("after inserts: count={d} leaves={d} depth={d} nodes={d}\n", .{ ix.count(), ix.leaf_count, ix.depth, ix.nodes.items.len }); + std.debug.print("after inserts: count={d} leaves={d} depth={d} nodes={d}\n", .{ + ix.count(), + ix.leaf_count, + ix.depth, + ix.nodes.items.len, + }); if (ix.count() != N + M) return error.BadCount; for (0..500) |_| { const a = rand.intRangeAtMost(i32, 0, 99); @@ -113,7 +137,12 @@ pub fn main() !void { ix.remove_doc(gpa, d, ids.items[i]); alive.items[i] = false; } - std.debug.print("after deletes: count={d} leaves={d} depth={d} nodes={d}\n", .{ ix.count(), ix.leaf_count, ix.depth, ix.nodes.items.len }); + std.debug.print("after deletes: count={d} leaves={d} depth={d} nodes={d}\n", .{ + ix.count(), + ix.leaf_count, + ix.depth, + ix.nodes.items.len, + }); if (ix.count() != (N + M) - (N + M) / 3 - 1) return error.BadCount; for (0..500) |_| { const a = rand.intRangeAtMost(i32, 0, 99); @@ -131,9 +160,15 @@ pub fn main() !void { defer out.deinit(gpa); try ix.lookup_eq(gpa, &.{.{ .int32 = a }}, &out); var expected: usize = 0; - for (facts.items, 0..) |f, fi| { if (alive.items[fi] and f.a == a) expected += 1; } + for (facts.items, 0..) |f, fi| { + if (alive.items[fi] and f.a == a) expected += 1; + } if (out.items.len != expected) { - std.debug.print("EQ MISMATCH a={d}: got {d} want {d}\n", .{ a, out.items.len, expected }); + std.debug.print("EQ MISMATCH a={d}: got {d} want {d}\n", .{ + a, + out.items.len, + expected, + }); return error.BadCount; } } @@ -154,11 +189,19 @@ pub fn main() !void { ix.remove_doc(gpa, d, ids.items[i]); remaining -= 1; if (ix.count() != remaining) { - std.debug.print("COUNT MISMATCH during drain: {d} != {d}\n", .{ ix.count(), remaining }); + std.debug.print("COUNT MISMATCH during drain: {d} != {d}\n", .{ + ix.count(), + remaining, + }); return error.BadCount; } } - std.debug.print("after full drain: count={d} leaves={d} depth={d} nodes={d}\n", .{ ix.count(), ix.leaf_count, ix.depth, ix.nodes.items.len }); + std.debug.print("after full drain: count={d} leaves={d} depth={d} nodes={d}\n", .{ + ix.count(), + ix.leaf_count, + ix.depth, + ix.nodes.items.len, + }); if (ix.count() != 0) return error.BadCount; // The drained tree still accepts and finds entries. pairs[0] = .{ .key = "a", .value = .{ .int32 = 7 } }; diff --git a/src/update.zig b/src/update.zig index 447fc62..99b90ba 100644 --- a/src/update.zig +++ b/src/update.zig @@ -21,7 +21,12 @@ pub fn apply(doc: *bson.Document, update: *const bson.Document) UpdateError!void doc.pairs = try pairs.toOwnedSlice(arena); } -fn apply_operator(arena: std.mem.Allocator, pairs: *std.ArrayListUnmanaged(bson.Pair), op: []const u8, value: bson.Value) UpdateError!void { +fn apply_operator( + arena: std.mem.Allocator, + pairs: *std.ArrayListUnmanaged(bson.Pair), + op: []const u8, + value: bson.Value, +) UpdateError!void { if (std.mem.eql(u8, op, "$set")) { const ops = doc_pairs(value) orelse return error.InvalidUpdate; for (ops) |p| { @@ -146,7 +151,11 @@ fn parse_index(seg: []const u8) ?usize { } /// Shallow-copy a slice into a growable list backed by `arena`. -fn copy_to_list(comptime T: type, arena: std.mem.Allocator, items: []const T) UpdateError!std.ArrayListUnmanaged(T) { +fn copy_to_list( + comptime T: type, + arena: std.mem.Allocator, + items: []const T, +) UpdateError!std.ArrayListUnmanaged(T) { var out: std.ArrayListUnmanaged(T) = .empty; errdefer out.deinit(arena); try out.appendSlice(arena, items); @@ -173,7 +182,12 @@ fn get_value(pairs: []const bson.Pair, segs: []const []const u8) ?bson.Value { }; } -fn set_path(arena: std.mem.Allocator, pairs: *std.ArrayListUnmanaged(bson.Pair), segs: []const []const u8, value: bson.Value) UpdateError!void { +fn set_path( + arena: std.mem.Allocator, + pairs: *std.ArrayListUnmanaged(bson.Pair), + segs: []const []const u8, + value: bson.Value, +) UpdateError!void { if (segs.len == 1) { if (find_pair(pairs.items, segs[0])) |idx| { pairs.items[idx].value = value; @@ -237,7 +251,11 @@ fn set_path(arena: std.mem.Allocator, pairs: *std.ArrayListUnmanaged(bson.Pair), } } -fn unset_path(arena: std.mem.Allocator, pairs: *std.ArrayListUnmanaged(bson.Pair), segs: []const []const u8) void { +fn unset_path( + arena: std.mem.Allocator, + pairs: *std.ArrayListUnmanaged(bson.Pair), + segs: []const []const u8, +) void { if (segs.len == 1) { if (find_pair(pairs.items, segs[0])) |idx| { _ = pairs.orderedRemove(idx); diff --git a/src/wire.zig b/src/wire.zig index ad8da0f..7b788c8 100644 --- a/src/wire.zig +++ b/src/wire.zig @@ -256,7 +256,13 @@ pub const Reply = struct { } /// Serialize this reply as a full OP_MSG message. - pub fn build(self: *Reply, gpa: std.mem.Allocator, request_id: u32, response_to: u32, out: *std.ArrayListUnmanaged(u8)) !void { + pub fn build( + self: *Reply, + gpa: std.mem.Allocator, + request_id: u32, + response_to: u32, + out: *std.ArrayListUnmanaged(u8), + ) !void { try write_message(gpa, request_id, response_to, 0, self.pairs.items, out); } };