Skip to content
16 changes: 9 additions & 7 deletions .github/actions/rust-cache/action.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: Rust cache
description: Install mbx and its Cargo shim, using local storage in runner VMs and R2 on hosted runners.
description: Install mbx and its Cargo shim, using local storage on self-hosted runners and R2 on hosted runners.

inputs:
access-key-id:
Expand Down Expand Up @@ -88,19 +88,21 @@ runs:
remote-url: s3://mbx-moq-dev
namespace: ${{ runner.os }}-${{ runner.arch }}

# A runner VM keeps managed targets and learned incremental state across
# jobs, and Cargo trusts whatever outputs it finds there. State another mbx
# version left is not trusted, since 1.26's was corrupt, so it goes before
# the first build on a new version. The shared store stays, so that build
# restores what it needs rather than compiling it.
# A self-hosted runner keeps managed targets and learned incremental state
# across jobs, and Cargo trusts whatever outputs it finds there. State
# another mbx version left is not trusted, since 1.26's was corrupt, so it
# goes before the first build on a new version. The shared store stays, so
# that build restores what it needs rather than compiling it. Runners on one
# host may share the home but never a workspace, so each workspace has its
# own stamp.
- name: Drop another mbx version's private state
if: runner.environment == 'self-hosted'
shell: bash
env:
VERSION: ${{ steps.mbx.outputs.mbx-version }}
run: |
: "${VERSION:?the mbx action reported no version}"
stamp="$HOME/.cache/moq-mbx-version"
stamp="$HOME/.cache/moq-mbx-versions$RUNNER_WORKSPACE"
if [ "$(cat "$stamp" 2>/dev/null)" != "$VERSION" ]; then
mbx clean --under "$RUNNER_WORKSPACE"
mkdir -p "${stamp%/*}"
Expand Down
5 changes: 3 additions & 2 deletions .github/workflows/check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -75,8 +75,9 @@ jobs:
access-key-id: ${{ secrets.MBX_R2_ACCESS_KEY_ID }}
secret-access-key: ${{ secrets.MBX_R2_SECRET_ACCESS_KEY }}

# The pre-setup query and execution use the same impact map. A runner VM
# keeps target/, so drop the previous job's build timings first.
# The pre-setup query and execution use the same impact map. A
# self-hosted runner keeps target/, so drop the previous job's build
# timings first.
- name: Check
if: steps.scope.outputs.needed == 'true'
run: |
Expand Down
21 changes: 12 additions & 9 deletions quest/m0/ci/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,15 +53,18 @@ Decisions (maintainer, 2026-10-13):
- Self-hosted CI is dropped: hosted runners plus the R2 caches replace the
spare-desktop runner, its cache server, and the cache release it waited on.
Reopened 2026-10-19, conditionally, to compare runners once the pipeline
was optimized. Decided 2026-10-22 (maintainer): the queue runner is the
persistent `nix/runner` VM (no host mounts or tailnet, firewalled to public
egress), in use since 2026-10-20. Over 32 one-pass entries it took a median
5.5 minutes per merged PR (p90 11.9, at most 24.2) against 21 minutes on a
hosted runner, with no per-minute bill. Blacksmith is not pursued.
Open (found 2026-10-23): the runner deployed on that host is a
systemd-nspawn container (`container@moq-ci`) with `/dev/kvm` and a host
cache directory bound in, not the VM this decision and `nix/runner/README.md`
describe. Confirm which is intended before changing either.
was optimized. Decided 2026-10-22 (maintainer): the queue runs on a
persistent self-hosted runner, in use since 2026-10-20. Over 32 one-pass
entries it took a median 5.5 minutes per merged PR (p90 11.9, at most 24.2)
against 21 minutes on a hosted runner, with no per-minute bill. Blacksmith
is not pursued.
- Decided 2026-10-25 (maintainer), superseding the 2026-10-22 choice of the
`nix/runner` VM: no VM. The queue runner is the deployed systemd-nspawn
container `container@moq-ci`, and new queue runners go into it. Reason: the
CI host should run mostly sandboxed through kernel primitives (namespaces,
cgroups and slices, seccomp), which the container already uses, so a VM is
not needed for isolation. The `nix/runner` VM module is not the deployment
target.

Decided 2026-10-23 (maintainer), to speed up the serial queue: structural
changes first (concurrent entries, parallel modules, one test build), then a
Expand Down
56 changes: 44 additions & 12 deletions quest/m0/ci/queue-concurrency.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# [M] Build queue entries concurrently
# [S] Build queue entries concurrently

## Goal

Expand All @@ -15,16 +15,48 @@ Measured 2026-10-23 over 32 merge-queue runs: `max_entries_to_build: 1` with one
`moq-queue` runner makes the queue fully serial, about 7 merged PRs per hour,
with a median entry of 135 s (p90 516 s) and 3 to 12 s of runner wait.

- Add queue runner capacity (another instance or container) and raise
`max_entries_to_build` to match. The host also runs the `moq-pro` runner
under the same `ci.slice` quota and a desktop; split CPU and memory so
concurrent entries do not slow each other past the gain.
- Concurrent entries share the local mbx store; confirm that is safe, or give
each its own. Each job sweeps the store before it builds
(`.github/actions/rust-cache`), which can evict generated source trees that
a concurrent entry's next build reads.
- Measure merged PRs per hour and per-entry wall time before and after.
- Queue entries skip `quest gate`: `sh/local.sh` execs directly under GitHub
Actions, so the runner quota, not admission, has to keep entries apart.
Decided 2026-10-25 (maintainer): no VM. The second runner goes into the
deployed `container@moq-ci` systemd-nspawn container, sandboxed by kernel
primitives (see the [CI epic](/quest/m0/ci/README.md)), not into the
`nix/runner` VM module. The container is defined in the host's own flake, not
in this repository, so registering the runner is a host change.

Each runner keeps its own work directory, since the nixpkgs runner service
empties it on start. The self-hosted cache step keeps its mbx version stamp per
workspace, so a runner sharing a home with another still drops its own
workspace's state after an mbx version change.

Decided 2026-10-25 (maintainer): ✅ one home under one static user for both
runners. Concurrent entries share mbx's store, wait on a compilation the other
entry is running instead of repeating it, and draw on one machine-wide compile
budget, which mbx documents as supported
([how it works](https://github.com/jdx/mr-boxington/blob/v1.25.0/docs/how-it-works.md#machine-wide-scheduling)).
nixpkgs gives each runner its own `DynamicUser` by default, so both need the
same `user` and `HOME`. Rejected: a home per runner (two cold caches, double
the disk, two mbx schedulers each budgeting the whole container's CPUs) and a
second container (those costs plus another container). A NixOS container
builds through the host's Nix daemon by default, so the Nix store is shared
either way.

Remaining (maintainer, on the host):

- Register a second runner in `container@moq-ci` with its own name and work
directory, under the same static user and `HOME` as the first, with a fresh
registration token.
- Size the container's share of `ci.slice`, which it shares with the `moq-pro`
runner and a desktop, so concurrent entries do not slow each other past the
gain. Queue entries skip `quest gate` (`sh/local.sh` execs directly under
GitHub Actions), so the cgroup quota, not admission, keeps them apart.
- Keep `NEXTEST_TEST_THREADS=24` per entry (48 workers across two entries;
mbx budgets compiles, not tests) and watch the first day for timing flakes
rather than guess a lower number.
- Make the pre-build mbx sweep safe for a concurrent entry first. With
[moq-dev/moq#6709](https://github.com/moq-dev/moq/pull/6709), each job runs
`mbx gc` before it builds (`.github/actions/rust-cache`), and in a shared
home that sweep can evict generated source trees another entry's next
build reads.
- Raise the main ruleset's `max_entries_to_build` from 1 to 2.
- Measure merged PRs per hour and per-entry wall time for a day against the
baseline before going past 2.

Public API: none. Wire: none.
2 changes: 1 addition & 1 deletion quest/m2/audit-ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ verdicts:
ran). The longest 2026-10-22 jobs: Audio quality 62 minutes, Loom 60
(cancelled), binding benchmark 52, Rust bench targets 48, relay profiles 45,
Rust benchmark tests 44, the gate suite 40.
- The merge queue runs one `just ci` job on the persistent `nix/runner` VM:
- The merge queue runs one `just ci` job on the self-hosted `moq-queue` runner:
median 5.5 minutes, p90 11.9, at most 24.2 over 32 entries (m0, 2026-10-21).
Its job summary lists seconds per module, and the `cargo-timings` artifact
is the per-crate drill-down.
Expand Down
5 changes: 5 additions & 0 deletions quest/m2/gpu-ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,11 @@ skipping inside the Nix shell.
guest, and keep the runner registered only to this repository with
approval required for every external contributor's workflow, as the
README requires.
- Open (2026-10-25): the [CI epic](/quest/m0/ci/README.md) dropped the VM
for the CI host in favor of kernel sandboxing, which conflicts with VFIO
passthrough. Recommendation: a container runner with the NVIDIA device
nodes and the host driver's libraries bound in. Confirm with the
maintainer before starting.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

Public API: none. Wire: none.

Expand Down
Loading