Skip to content

spike: per-crate Nix builds (crate2nix) against crane - #8662

Closed
kixelated wants to merge 3 commits into
mainfrom
spike/crate2nix
Closed

kixelated wants to merge 3 commits into
mainfrom
spike/crate2nix

Conversation

@kixelated

Copy link
Copy Markdown
Collaborator

Spike, not for landing: would one Nix derivation per crate (crate2nix's generated Cargo.nix on nixpkgs' buildRustCrate) make caching of moq's Nix-built Rust meaningfully better than crane, with nix.moq.dev's CI bucket as the shared cache? just check and just test were not run: this branch will not land.

Problem

Today Nix builds Rust in one place per main push: Cache's runtime job (sh/gh/cache-runtime.sh, GitHub-hosted ubuntu-24.04) builds crane's dependency layers (.#moq-relay.cargoArtifacts, .#moq.cargoArtifacts, now also moq-boy's) and the packages, then pushes them to nix-ci-moq-dev (14-day expiry, refresh restarts the age of the newest cached commit's closure). Smoke's nix cell and deploys substitute the packages of the newest cached commit. Release Nix builds tagged packages on three platforms into nix-moq-dev, and macOS release dry-runs build .#<bin> through Nix. Everything else (just ci: clippy, nextest, every feature set and platform) builds with cargo through mbx, cached in the mbx-moq-dev R2 bucket.

crane caches only two layers. Its package derivation takes the whole filtered workspace, so any .rs change anywhere rebuilds every workspace crate of both packages. Its dependency layer takes Cargo.lock, so any lock change, release-plz version bumps included, rebuilds all ~430 dependencies. Last week 606 Cache runs started; 380 of their runtime jobs ran 8 to 22 minutes (rebuilding), the rest 1 to 2.

Approach

  • nix/Cargo.nix (crate2nix 0.15.0) and nix/crate2nix.nix build moq-relay and moq-cli per crate, on the same rust-overlay 1.98.1 toolchain crane uses, with [profile.release] restated: codegen-units 1, fat LTO and strip=symbols on the root binaries only, opt-level 0 build scripts and proc macros, frame pointers on every crate, crane's source filter on workspace crates. Features resolve per package, as crane's two -p builds do.
  • spike-crate2nix matches the per-crate layout a downstream spike used: every crate emits machine code and bitcode. spike-crate2nix-bitcode compiles every crate outside the host closure (proc macros, build-script dependencies, and their dependencies) with -C linker-plugin-lto, bitcode only, as cargo does under fat LTO: 215 of 539 derivations.
  • spike-crane is today's two packages with a times hook; every variant prints builder CPU into nix log. nix/spike/ holds the scripts.

Measurements

Desktop, 24 cores, Nix 2.35.2, one build at a time through quest gate, --max-jobs 24. CPU is user+sys summed over every derivation built. Warm: the base commit's outputs are in the local store. Subst: the base's Rust outputs are deleted locally and served, signed, from a file:// cache, as nix.moq.dev would. Edits: a comment appended to rs/moq-relay/src/main.rs (leaf) and rs/moq-net/src/lib.rs (core); lock: cargo update -p arc-swap -p flate2 -p itoa.

Scenario crane CPU s / wall s per-crate per-crate, bitcode deps Rebuilt (crane / per-crate)
cold 1526 / 404 (rerun 1529 / 403) 1729 / 636 (load 15 to 23) 1359 / 414 5 / 539 drvs
leaf, warm 777 / 329 396 / 268 376 / 251 both packages, every workspace crate / 3 of 539 crates
leaf, subst 729 / 310 406 / 274
core, warm 776 / 381 818 / 434 686 / 371 same / 28 of 539
core, subst 703 / 302 847 / 455
lock, warm 1484 / 378 931 / 442 ~830 / 419 everything / 98 of 539
lock, subst 1480 / 366 934 / 445
  • Bitcode lock: 763 CPU-s measured plus 67 for 37 host crates the previous column had already built.
  • The final link dominates edit wall time: moq-cli's fat-LTO build is ~236 CPU-s on one core in every variant.

Per change, what a CI nix copy would add to the bucket, and what a fresh runner downloads:

Change crane push per-crate push bitcode push crane download per-crate download
leaf 3 paths, 27 MiB 7 paths, 45 MiB 7 paths, 44 MiB 546 MiB in 2 paths + 100 MiB vendored sources 384 MiB in 1054 paths
core 3 paths, 27 MiB 57 paths, 153 MiB 57 paths, 133 MiB same 252 MiB in 1000 paths
lock 10 paths, 574 MiB 197 paths, 201 MiB 197 paths, 179 MiB nothing reusable 205 MiB in 862 paths

Sizes are zstd, as just infra nix push writes. Each path is 2 objects (narinfo and NAR). A full per-crate set is 1077 paths, 405 MiB. crane's is 2 layers (573 MiB), 2 packages, and 910 vendored-source paths (100 MiB).

Weekly

Applying each of the 582 main commits from Nov 6 to 12 to the per-crate graph, priced from the measured per-derivation CPU. The model predicts 378, 682 and 785 CPU-s for the three scenarios above (measured 376, 686, ~830):

crane per-crate, bitcode deps
commits that build 260 (222 source, 38 lock or manifest) 210 (50 touch no crate either package uses)
builder CPU 228,700 s 116,100 s (-49%, ~31 CPU-hours)
paths pushed ~1,050 (~2,100 Class A puts) ~8,460 (~16,900 Class A puts)
bytes pushed 27.2 GiB 20.2 GiB

These are upper bounds, since Cache coalesces pushes. At R2's list prices both stay under a dollar a month: 16,900 puts a week is ~$0.08, inside the free 1M a month, and 14 days of pushes is ~40 GiB against ~54 GiB, $0.60 against $0.80 a month. refresh walks runtime closures, which hold no rlibs, so per-crate would record every crate output as a root: up to ~2,200 copy operations a week. crane's closure already refreshes ~1,850 objects, its vendored sources.

Fidelity

  • Binaries match crane's: identical --help, dynamic libraries and jemalloc, frame pointers kept (21249 against 21202 push %rbp), and sizes within 0.6%.
  • crate2nix mis-resolves features: it reads dep:rustls as also enabling a crate's same-named rustls feature. In reqwest 0.13 that feature pulls in aws-lc-rs, which is not in Cargo.lock, so rustls-webpki fails to compile. The fix keeps only features cargo resolved somewhere, read from crate2nix's internal.crates, an unstable interface.
  • Duplicated crates: per-package features give ~430 crates 539 derivations. tikv-jemalloc-sys builds twice (~97 CPU-s each), as it does in crane's two layers.
  • Stray toolchain: nixpkgs' default proc-macro-crate override references nixpkgs' cargo 1.99, not 1.98.1, which drags rustc 1.99 and LLVM (~600 MiB) into the closure once.
  • Measurement quirk: Nix 2.35 called the tikv-jemalloc-sys outputs alive with no root to name, so the subst runs kept them local. They are unchanged in every scenario.

Maintenance

  • Generated files: nix/Cargo.nix is 42,875 lines (1.6 MB), plus crate-hashes.json for the uniffi git fork. crate2nix generate takes 2.5 s. Any lock or manifest change makes it stale: 38 of last week's 582 main commits, release-plz bumps included. The lock bump here changed 12 lines.
  • CI freshness check: regenerate and diff on every PR (cargo and crate2nix, no build). Contributors without Nix need a bot that commits the result, or a patch they can apply, as the dropped quest planned.
  • Overrides to carry: jemalloc's fortify, the feature intersection, the host-closure scan, root-only LTO through EXTRA_LINK_ARGS_BINS, crateBin = [ ] on non-roots, and every future [profile] change restated in Nix.
  • Build settings: builds need --max-jobs, since one derivation at a time leaves most cores idle.

mbx

Per-crate Nix neither replaces nor duplicates mbx. mbx caches cargo's dev, test and clippy compilations for just ci, across feature sets and platforms. buildRustCrate compiles release LTO artifacts with its own -C metadata and path remaps, so no compilation is shared between the two. Tests stay on cargo and nextest. Per-crate Nix replaces only crane.

Downstream substitution

A consumer substitutes moq's crates only when it evaluates byte-identical derivations: the same nixpkgs (stdenv, buildRustCrate, its overrides), the same toolchain store path (moq's rust-overlay stable."1.98.1".default, not one derived from rust-toolchain.toml), the same Cargo.lock resolution, features, flags and source filter.

  • Its own workspace: a consumer that builds moq's crates in its own workspace has its own lock, features unified across its own roots, and its own profile. It shares effectively none of them, so it keeps its own generated Cargo.nix.
  • moq's flake outputs: a consumer that evaluates moq's flake outputs shares everything. It already substitutes the finished package today.
  • moq's own lockfile: between the two, a consumer building, say, an FFI library from moq's own lockfile could substitute crates if moq exported the builder and its CI built that same root with the same features. moq's CI builds nothing like that under Nix today.

Recommendation

Not worth it.

  • The win: per-crate Nix with bitcode-only dependencies is 11% cheaper cold and 52% cheaper on a leaf edit than crane, and modeled 49% (~112,600 CPU-s a week) cheaper on Cache's runtime job. That removes the cost objection behind the decision in moq-dev/moq#6886.
  • Where it lands: those CPU-seconds run on free GitHub-hosted runners. R2 costs stay under a dollar a month either way, with more puts and fewer bytes. Hosts and Smoke already substitute finished packages for cached commits, and a downstream workspace cannot substitute moq's crates.
  • What it costs: a second feature resolver that mis-resolved on the first build, a dependence on crate2nix internals, a generated file that 38 commits a week must regenerate, and a restated release profile.
  • If that changes: if the Cache job's latency or a consumer's builds of uncached trees start to matter, this branch's bitcode variant is the shape to adopt.

Impact

None. The spike adds flake outputs only.

Alternatives

  • crane with mbx in the sandbox: ruled out in moq-dev/moq#6812, since the Nix sandbox cannot reach R2.
  • Machine code for every crate: the downstream layout, the spike-crate2nix column. It costs 13% more than crane cold (under higher load) and 5% more on the core edit.

Follow-ups

  • flake.nix's header says CI paths expire after 30 days. infra/nix/lifecycle.json expires them after 14. Recommend planning a one-line fix.

Models

Size: M
Implementer: Claude Code, claude-opus-5-5, high effort

🤖 Generated with Claude Code

(Written by Opus 5.5)

kixelated and others added 3 commits November 13, 2026 09:35
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Superseded by moq-dev/moq#8666, which lands this spike's bitcode-only per-crate layout for every Nix-built Rust package and removes crane, following the maintainer's 2026-11-13 decision to adopt it. Closing this spike.

(Written by Opus 5.5)

@kixelated kixelated closed this Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant