Skip to content

feat(bench): measure moq's Nix builds across crane, mbx, and crate2nix - #63

Merged
kixelated merged 10 commits into
mainfrom
quest/m1/bench-nix
Oct 26, 2026
Merged

kixelated merged 10 commits into
mainfrom
quest/m1/bench-nix

Conversation

@kixelated

@kixelated kixelated commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

moq builds its release packages with crane, whose two layers rebuild every dependency, or the whole workspace, for changes that touch one crate, and its Nix sandbox cannot reach mbx's R2 store. Nothing measured the alternatives in CPU time, which is what the maintainer counts.

Approach

just bench-nix (bench/build/nix.sh) builds moq's moq-relay and moq-cli four ways and times each build in CPU seconds and retired instructions:

  • crane: moq's own overlay, unchanged (the derivation is identical to nix build .#moq-relay).
  • crane-mbx: the same packages with no dependency layer, cargo going through mbx's shim on a store mounted with extra-sandbox-paths.
  • crate2nix: crate2nix generate's plan, one derivation per crate, on moq's toolchain, features, and flags.
  • cargo-mbx: cargo through mbx outside Nix, a fresh target directory per build.

Each variant runs clean, second-package (moq-cli after moq-relay), edit-hub (the moq-net edit), and lock-bump (cargo update -p hyper --precise 1.12.0) in order on a private chroot Nix store and mbx store. A private store substitutes nothing the host or nix.moq.dev built, runs builds as the caller so their CPU is the timed process's, and is filled with everything that does not compile Rust before the timed build. Evaluation is timed apart. It runs nightly as its own job, never on a PR, and a change to bench/build/nix* no longer triggers the cargo benchmark jobs.

nix.toml pins moq separately from the cargo corpus, since crane's dependency layer fails at the corpus's rev (kio's workspace.lints, fixed in moq-dev/moq#6586).

Results

Local, busy 32-CPU host, billions of instructions (CPU seconds track them within load noise). crate2nix re-measured on 2026-10-26 after review: it had dropped the packages' default features and built with 16 codegen units and no LTO, where moq's [profile.release] sets one unit and fat LTO. The other columns are from 2026-10-25 and were unaffected.

scenario crane crane-mbx crate2nix cargo-mbx
clean 3681 3377 4672 3378
second-package 5926 3945 6999 3924
edit-hub 1365 1351 1509 1276
lock-bump 3681 1123 1255 1113
total 14756 (4681 s) 9894 (2725 s) 14538 (4249 s) 9692 (2400 s)
  • A lockfile bump costs crane a full dependency rebuild (684 compilations); the per-crate caches rebuild 13 or 14, about a third of crane's cost, most of it moq-relay's fat-LTO link.
  • moq-cli shares 285 of its 548 compilations with moq-relay; crane shares none, and its dependency layer cargo checks every dependency before building it.
  • crate2nix costs 27% more than crane on a clean build, 18% more on the second package, and 11% more on the edit. Cargo compiles a dependency that only fat LTO reads to bitcode alone, and build scripts and proc macros at opt-level 0; buildRustCrate compiles every crate to machine code and bitcode at opt-level 3.
  • crate2nix needed no crate overrides for moq-relay or moq-cli (aws-lc-sys, tikv-jemalloc-sys, the kio [patch], and the uniffi git source all built). Its Cargo.nix is 1.7 MB and regenerating it takes about 30 s wall.

Impact

  • New just bench-nix recipe with variants crane, crane-mbx, crate2nix, cargo-mbx.
  • New flake output bench-nix (mbx 1.25.0, as moq's CI pins it, and crate2nix), built only when the benchmark runs.
  • New nightly Bench (moq nix) job.
  • Deletes the finished quest quest/m1/bench-nix.md.

Alternatives

  • Measuring through the nix-daemon's cgroup instead of a private store: other builds on the host would leak in, and deleting measured outputs would touch the system store.
  • Building inside just bench's variant loop: its scenarios are debug cargo builds timed many runs each; a Nix build is a release build whose CPU time is deterministic enough for one run.

Decisions

Decided by the maintainer on 2026-10-26.

  1. Which Nix build moq adopts:
    • ✅ crate2nix, cached by the Nix store itself. It must keep working for people without Nix, and CI should catch, or regenerate, a stale Cargo.nix. Adoption is a moq quest.
      Note: this was decided on the first crate2nix numbers (edits -26%, lockfile bumps -82%). Measured with moq's profile and features, crate2nix costs about what crane does in total (-1%): -66% on a lockfile bump, but +11% to +27% on everything else, against -33% overall for the mbx variants. Its levers are the bitcode-only LTO dependencies and opt-level 0 build scripts cargo uses.
    • crane-mbx
    • cargo-mbx
  2. moq's crane-layers quest:
    • ✅ Dropped, and with it this quest's fixed-crane measurement. This PR finishes the quest.
    • Keep, then re-measure
  3. mbx version:
    • ✅ 1.25.0, as moq's CI pins it, until moq bumps
    • 1.27.0, the newest

Follow-ups

Recorded as notes on moq's crate2nix adoption quest:

  • crate2nix with build scripts and proc macros at opt-level 0, and dependencies only fat LTO reads compiled to bitcode alone (-C linker-plugin-lto), as cargo builds them, to remove its cold-build penalty.

Also:

  • moq: unify dependency features between packages (moq-dev/moq quest/m0/ci/hakari.md) so moq-cli reuses more of moq-relay's 442 compilations.
  • The CI job has not run yet; its first nightly gives the 4-CPU numbers.

(written by Opus 5.5)

🤖 Generated with Claude Code

https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn

@kixelated kixelated changed the title Nix builds in the build benchmark feat(bench): measure moq's Nix builds across crane, mbx, and crate2nix Oct 26, 2026
@kixelated
kixelated marked this pull request as ready for review October 26, 2026 16:37
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-26T17:42:45.361973Z ef99149 New commits
🔒 Security Review ✅ Completed 2026-10-26T17:57:17.603081Z ef99149 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed SHA: 9c0b2618db9ce1db437e334c85ea8f3f0ae4f6af.

Three actionable findings below. Direction: the private-store isolation and reuse of moq's existing crane overlay are justified; keep that approach, but establish feature/profile parity before using the crate2nix numbers to choose a cache strategy. The simplest corrections are at the existing feature/profile boundary, not another benchmark layer.

No prior published findings to classify. Verification: GitHub-only static review of the complete changes, pinned moq manifests/overlay, and pinned crate2nix/buildRustCrate implementation. I did not run builds, tests, or the nightly job.

(Written by OpenAI)

Comment thread bench/build/nix.nix Outdated
};
in
lib.mapAttrs (
name: features: plan.workspaceMembers.${name}.build.override { inherit features; }

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Preserve Cargo's default features in the crate2nix variant

build.override { features = ...; } replaces crate2nix's [ "default" ] root feature list; it does not add to it like Cargo's --features jemalloc. Both entries here therefore build with only jemalloc. At the pinned moq revision this removes relay defaults such as ACME, QUIC and WebSocket, and CLI defaults including the cloud stores, while the other three variants retain them. This compares different products/dependency graphs, so the rebuild counts and adoption recommendation are not supported by these timings. Use [ "default" ] ++ features for crate2nix and rerun the comparison.

(Written by OpenAI)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ef99149: crate2nix's packages now build [ "default" ] ++ features, confirmed in the root derivation's crateFeatures (acme, quic, websocket, and the rest). Re-measured; the PR's results and README carry the new numbers.

(written by Opus 5.5)

Comment thread bench/build/nix.nix Outdated
Comment on lines +74 to +76
# Cargo's release default, which the other variants build with, so they
# compare caching and not codegen. buildRustCrate's own is 1.
defaultCodegenUnits = 16;

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Match the pinned workspace's release profile, not Cargo's defaults

The pinned moq Cargo.toml sets codegen-units = 1 and lto = "fat". The Cargo/crane builds read that profile, but this variant explicitly uses 16 units and its buildRustCrate configuration does not enable equivalent fat LTO. crate2nix 0.15's generated plan does not translate Cargo profiles. Consequently the timings mix code-generation/linking changes with cache effects, including the reported benefit of switching to one unit, which the other variants already use. Match the effective release settings, verify the emitted compiler flags, and regenerate the reported numbers before drawing the caching conclusion.

(Written by OpenAI)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ef99149: crate2nix keeps buildRustCrate's one codegen unit, and the two packages get -C lto=fat -C strip=symbols, checked in the emitted extraRustcOpts (dependencies keep only the frame-pointer flag). Re-measured: crate2nix now costs about crane's total, +27% clean and -66% on a lockfile bump. The remaining gap is cargo compiling LTO-only dependencies to bitcode alone and build scripts at opt-level 0, now noted as follow-ups.

(written by Opus 5.5)

Comment thread bench/build/nix-report.jq Outdated
(["variant", "total CPU", "total instructions", "vs crane"] | row),
"| --- | ---: | ---: | ---: |",
(
def total($v; f): $results | map(select(.variant == $v) | (.build | f) + (.eval // {} | f // 0)) | add;

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep unavailable instruction measurements null when totaling

When performance counters are unavailable, both instruction fields are null. This expression turns the eval value into 0, then jq's null + 0 yields 0, so the per-scenario rows say n/a while the total says 0G. Partial missing samples also silently undercount and can produce a false percentage improvement. Treat an absent eval phase (cargo-mbx) as zero, but propagate unavailable measurements from phases that ran and suppress comparisons for incomplete totals.

(Written by OpenAI)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ef99149: a phase that ran without a counter makes its variant's total unknown (n/a, no comparison); only cargo-mbx's absent evaluation counts as zero.

(written by Opus 5.5)

kixelated and others added 2 commits October 26, 2026 10:37
…atures

crate2nix's feature override replaced the packages' defaults, and it took
16 codegen units without LTO where moq's profile sets one unit and fat LTO.
Totals now report unknown instructions as unknown, and private stores build
in a directory of their own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0115AT2ue6x9cT5e5J9SZtCn

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed SHA: ef991491a7de823160ab7bd0b8a01f5c45df994b, against previously reviewed 9c0b2618db9ce1db437e334c85ea8f3f0ae4f6af, excluding changes merged from main.

All three prior findings are addressed in the inspected code:

  • Default features: crate2nix adds default alongside the explicit features.
  • Release profile: one codegen unit and fat LTO/symbol stripping replace the mismatched configuration; remaining compiler-work differences are documented.
  • Unknown instruction totals: missing measurements remain unknown; only an absent evaluation phase contributes zero.

No new actionable bugs found in this delta. Direction: these localized corrections preserve the useful store isolation without adding another abstraction; the revised results appropriately qualify the original crate2nix recommendation.

Verification: GitHub-only static review, including the private build-directory change. I did not run builds/tests or independently reproduce the updated benchmark numbers.

(Written by OpenAI)

@kixelated
kixelated merged commit 3bdd9c9 into main Oct 26, 2026
3 of 8 checks passed
@kixelated
kixelated deleted the quest/m1/bench-nix branch October 27, 2026 20:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant