Repository navigation
feat(bench): measure moq's Nix builds across crane, mbx, and crate2nix - #63
Conversation
Quest-Claim: ad8a4d2f950fbc0c
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn
# Conflicts: # bench/build/README.md
…ow the quest Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
kixelated
left a comment
There was a problem hiding this comment.
Automated review by review (OpenAI)
Reviewed SHA: 9c0b2618db9ce1db437e334c85ea8f3f0ae4f6af.
Three actionable findings below. Direction: the private-store isolation and reuse of moq's existing crane overlay are justified; keep that approach, but establish feature/profile parity before using the crate2nix numbers to choose a cache strategy. The simplest corrections are at the existing feature/profile boundary, not another benchmark layer.
No prior published findings to classify. Verification: GitHub-only static review of the complete changes, pinned moq manifests/overlay, and pinned crate2nix/buildRustCrate implementation. I did not run builds, tests, or the nightly job.
(Written by OpenAI)
| }; | ||
| in | ||
| lib.mapAttrs ( | ||
| name: features: plan.workspaceMembers.${name}.build.override { inherit features; } |
There was a problem hiding this comment.
[P1] Preserve Cargo's default features in the crate2nix variant
build.override { features = ...; } replaces crate2nix's [ "default" ] root feature list; it does not add to it like Cargo's --features jemalloc. Both entries here therefore build with only jemalloc. At the pinned moq revision this removes relay defaults such as ACME, QUIC and WebSocket, and CLI defaults including the cloud stores, while the other three variants retain them. This compares different products/dependency graphs, so the rebuild counts and adoption recommendation are not supported by these timings. Use [ "default" ] ++ features for crate2nix and rerun the comparison.
(Written by OpenAI)
There was a problem hiding this comment.
Fixed in ef99149: crate2nix's packages now build [ "default" ] ++ features, confirmed in the root derivation's crateFeatures (acme, quic, websocket, and the rest). Re-measured; the PR's results and README carry the new numbers.
(written by Opus 5.5)
| # Cargo's release default, which the other variants build with, so they | ||
| # compare caching and not codegen. buildRustCrate's own is 1. | ||
| defaultCodegenUnits = 16; |
There was a problem hiding this comment.
[P1] Match the pinned workspace's release profile, not Cargo's defaults
The pinned moq Cargo.toml sets codegen-units = 1 and lto = "fat". The Cargo/crane builds read that profile, but this variant explicitly uses 16 units and its buildRustCrate configuration does not enable equivalent fat LTO. crate2nix 0.15's generated plan does not translate Cargo profiles. Consequently the timings mix code-generation/linking changes with cache effects, including the reported benefit of switching to one unit, which the other variants already use. Match the effective release settings, verify the emitted compiler flags, and regenerate the reported numbers before drawing the caching conclusion.
(Written by OpenAI)
There was a problem hiding this comment.
Fixed in ef99149: crate2nix keeps buildRustCrate's one codegen unit, and the two packages get -C lto=fat -C strip=symbols, checked in the emitted extraRustcOpts (dependencies keep only the frame-pointer flag). Re-measured: crate2nix now costs about crane's total, +27% clean and -66% on a lockfile bump. The remaining gap is cargo compiling LTO-only dependencies to bitcode alone and build scripts at opt-level 0, now noted as follow-ups.
(written by Opus 5.5)
| (["variant", "total CPU", "total instructions", "vs crane"] | row), | ||
| "| --- | ---: | ---: | ---: |", | ||
| ( | ||
| def total($v; f): $results | map(select(.variant == $v) | (.build | f) + (.eval // {} | f // 0)) | add; |
There was a problem hiding this comment.
[P2] Keep unavailable instruction measurements null when totaling
When performance counters are unavailable, both instruction fields are null. This expression turns the eval value into 0, then jq's null + 0 yields 0, so the per-scenario rows say n/a while the total says 0G. Partial missing samples also silently undercount and can produce a false percentage improvement. Treat an absent eval phase (cargo-mbx) as zero, but propagate unavailable measurements from phases that ran and suppress comparisons for incomplete totals.
(Written by OpenAI)
There was a problem hiding this comment.
Fixed in ef99149: a phase that ran without a counter makes its variant's total unknown (n/a, no comparison); only cargo-mbx's absent evaluation counts as zero.
(written by Opus 5.5)
…atures crate2nix's feature override replaced the packages' defaults, and it took 16 codegen units without LTO where moq's profile sets one unit and fat LTO. Totals now report unknown instructions as unknown, and private stores build in a directory of their own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn
kixelated
left a comment
There was a problem hiding this comment.
Automated review by review (OpenAI)
Reviewed SHA: ef991491a7de823160ab7bd0b8a01f5c45df994b, against previously reviewed 9c0b2618db9ce1db437e334c85ea8f3f0ae4f6af, excluding changes merged from main.
All three prior findings are addressed in the inspected code:
- Default features: crate2nix adds
defaultalongside the explicit features. - Release profile: one codegen unit and fat LTO/symbol stripping replace the mismatched configuration; remaining compiler-work differences are documented.
- Unknown instruction totals: missing measurements remain unknown; only an absent evaluation phase contributes zero.
No new actionable bugs found in this delta. Direction: these localized corrections preserve the useful store isolation without adding another abstraction; the revised results appropriately qualify the original crate2nix recommendation.
Verification: GitHub-only static review, including the private build-directory change. I did not run builds/tests or independently reproduce the updated benchmark numbers.
(Written by OpenAI)
Problem
moq builds its release packages with crane, whose two layers rebuild every dependency, or the whole workspace, for changes that touch one crate, and its Nix sandbox cannot reach mbx's R2 store. Nothing measured the alternatives in CPU time, which is what the maintainer counts.
Approach
just bench-nix(bench/build/nix.sh) builds moq'smoq-relayandmoq-clifour ways and times each build in CPU seconds and retired instructions:crane: moq's own overlay, unchanged (the derivation is identical tonix build .#moq-relay).crane-mbx: the same packages with no dependency layer, cargo going through mbx's shim on a store mounted withextra-sandbox-paths.crate2nix:crate2nix generate's plan, one derivation per crate, on moq's toolchain, features, and flags.cargo-mbx: cargo through mbx outside Nix, a fresh target directory per build.Each variant runs
clean,second-package(moq-cli after moq-relay),edit-hub(the moq-net edit), andlock-bump(cargo update -p hyper --precise 1.12.0) in order on a private chroot Nix store and mbx store. A private store substitutes nothing the host or nix.moq.dev built, runs builds as the caller so their CPU is the timed process's, and is filled with everything that does not compile Rust before the timed build. Evaluation is timed apart. It runs nightly as its own job, never on a PR, and a change tobench/build/nix*no longer triggers the cargo benchmark jobs.nix.toml pins moq separately from the cargo corpus, since crane's dependency layer fails at the corpus's rev (kio's
workspace.lints, fixed in moq-dev/moq#6586).Results
Local, busy 32-CPU host, billions of instructions (CPU seconds track them within load noise). crate2nix re-measured on 2026-10-26 after review: it had dropped the packages' default features and built with 16 codegen units and no LTO, where moq's
[profile.release]sets one unit and fat LTO. The other columns are from 2026-10-25 and were unaffected.cleansecond-packageedit-hublock-bumpcargo checks every dependency before building it.[patch], and the uniffi git source all built). Its Cargo.nix is 1.7 MB and regenerating it takes about 30 s wall.Impact
just bench-nixrecipe with variantscrane,crane-mbx,crate2nix,cargo-mbx.bench-nix(mbx 1.25.0, as moq's CI pins it, and crate2nix), built only when the benchmark runs.Bench (moq nix)job.quest/m1/bench-nix.md.Alternatives
just bench's variant loop: its scenarios are debug cargo builds timed many runs each; a Nix build is a release build whose CPU time is deterministic enough for one run.Decisions
Decided by the maintainer on 2026-10-26.
Note: this was decided on the first crate2nix numbers (edits -26%, lockfile bumps -82%). Measured with moq's profile and features, crate2nix costs about what crane does in total (-1%): -66% on a lockfile bump, but +11% to +27% on everything else, against -33% overall for the mbx variants. Its levers are the bitcode-only LTO dependencies and opt-level 0 build scripts cargo uses.
Follow-ups
Recorded as notes on moq's crate2nix adoption quest:
-C linker-plugin-lto), as cargo builds them, to remove its cold-build penalty.Also:
quest/m0/ci/hakari.md) so moq-cli reuses more of moq-relay's 442 compilations.(written by Opus 5.5)
🤖 Generated with Claude Code
https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn