Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .config/nextest.toml
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ slow-timeout = { period = "60s", terminate-after = 3 }
io-uring = { max-threads = 4 }
v4l2 = { max-threads = 1 }

# Nightly's ignored benchmark tests (`just gh nightly rust-test`) keep each test's
# Nightly's ignored benchmark tests (`just gh nightly rust-test|rust-test-apps`) keep each test's
# printed tables in target/nextest/bench/junit.xml for the benchmark report.
[profile.bench.junit]
path = "junit.xml"
Expand Down
6 changes: 6 additions & 0 deletions .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,7 @@ jobs:
# build environment, so a benchmark reuses the dependencies its group already
# compiled. The Rust benchmarks outgrew one hour, so they split by harness:
# nextest's ignored benchmark tests and cargo's bench targets share few builds.
# The benchmark tests split again by layer, since fat LTO links each test binary.
# A paired group also builds and runs the previous nightly's commit beside this
# one on the same runner, which is what fails the job on a regression; the
# cross-night history on the benchmark site stays informational.
Expand All @@ -117,6 +118,11 @@ jobs:
artifact: bench-rust-test
artifact-path: target/bench/nightly/rust-test
benchmark: true
- name: Rust application benchmark tests
recipe: gh nightly rust-test-apps
artifact: bench-rust-test-apps
artifact-path: target/bench/nightly/rust-test-apps
benchmark: true
- name: Binding benchmark runners
recipe: gh nightly bindings
- name: Rust bench targets
Expand Down
2 changes: 1 addition & 1 deletion bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -842,7 +842,7 @@ The table includes these measurements:
`size_bytes` and `peak_bytes`, `mib_per_s`, and `*_packets` and `*_messages`
columns are measurements, and every other column is a workload dimension in each series
name, so print outcomes only in measured columns. The Rust benchmark tests
(`just gh nightly rust-test`) run in release builds, one at a time, under
(`just gh nightly rust-test` and `rust-test-apps`) run in release builds, one at a time, under
nextest's `bench` profile, which keeps each test's output in a JUnit report;
their series are named by package and test path, such as
`rust-test/moq-congestion/gcc/bench/benchmark_per_packet`. A test with no runnable
Expand Down
9 changes: 5 additions & 4 deletions quest/m0/ci/hakari-followups.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,8 @@ Public API: none. Wire: none.
cost is material, present the measured tradeoff to the maintainer; recommend
keeping the narrow final exclusions unless the measurements justify another
design. That job timed out on five of the six scheduled runs from
2026-11-05 to 11-10 and on every one since 11-07, so a complete run waits on
[it fitting its hour](/quest/m0/ci/nightly-rust-test-hour.md). That change
regroups the job's builds, so compare only the mbx summaries of builds whose
crates, features, and profile match the baseline's.
2026-11-05 to 11-10 and on every one since 11-07. It is now two jobs,
`Nightly (Rust benchmark tests)` and `Nightly (Rust application benchmark
tests)`, each one release build grouping several crates, so none of their
builds match a baseline build's crates and features; compare only builds
that do.
9 changes: 2 additions & 7 deletions quest/m0/ci/nightly-relay-session-hour.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,5 @@ with `Evicted`. The
completed the sweep in about 25 and 23 minutes (their jobs failed later, at
`Fail on paired regressions`). Find what grew between those sweeps and the
60-minute cancellations from 11-08, then trim it or split the group across
matrix entries, as
[the Rust benchmark tests quest](/quest/m0/ci/nightly-rust-test-hour.md) does
for its job. The sweep still covers both fan-out axes.

## Related

- [Rust benchmark tests fit their hour](/quest/m0/ci/nightly-rust-test-hour.md) - the same timeout, caused by builds
matrix entries, as the Rust benchmark tests did. The sweep still covers both
fan-out axes.
34 changes: 0 additions & 34 deletions quest/m0/ci/nightly-rust-test-hour.md

This file was deleted.

2 changes: 1 addition & 1 deletion quest/m1/audio-jitter-target/calibrate.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ Carried decisions:
[moq-dev/moq#8308](https://github.com/moq-dev/moq/pull/8308#issuecomment-6730322286)):
✅ budget `trimmed_ms_per_min` and `player_skips_per_min` (rejected:
budgeting accelerates as discarded audio), and ✅ the nightly runner's
`bench-playback` counts as the quiet-host run (rejected: naming another
playback benchmarks count as the quiet-host run (rejected: naming another
host).

The rule: the worst of seven clean nights times 1.25, plus the documented
Expand Down
14 changes: 3 additions & 11 deletions rs/justfile
Original file line number Diff line number Diff line change
Expand Up @@ -584,14 +584,10 @@ codspeed:
# replaces only its own packages' benches, so `cargo codspeed run` runs both.
cd rs/moq-auth && VALGRIND_BUG_494162=1 cargo codspeed build --locked -m simulation --bench jwt

# Held feed FETCH readiness across independent request and track counts.
# Run the tests the arguments select in a release build, printing their output; benchmarks add `--run-ignored only`.
[positional-arguments]
bench-feed-fetch *args:
bash sh/rs/nextest.sh cargo nextest run --locked --release -p moq-net --lib --run-ignored only -E 'test(benchmark_feed_fetch_wakes)' --success-output immediate "$@"

# Device-free mixer and resampler cost, swept over fanout, packetization, and layout.
bench-playback:
bash sh/rs/nextest.sh cargo nextest run --locked --release -p moq-audio --features playback --run-ignored only -E 'test(benchmark_mixer_callbacks) | test(benchmark_resampler) | test(benchmark_raw_audio)' --success-output immediate
bench-tests *args:
bash sh/rs/nextest.sh cargo nextest run --locked --release --success-output immediate "$@"

# Sweep retained RTMP chunk-stream IDs independently of message and chunk sizes.
[positional-arguments]
Expand All @@ -608,10 +604,6 @@ bench-relay-auth *args:
bench-stats *args:
bash sh/local.sh cargo bench --locked -p moq-stats --features bench --bench producer_tick --bench decode --bench aggregate -- "$@"

# Sweep latency observations against writer count and report density.
bench-latency:
bash sh/rs/nextest.sh cargo nextest run --locked --release -p moq-bench --run-ignored only -E 'test(benchmark_observation_contention)' --success-output immediate

# Exact TS receiver scheduling across PID and queued-unit fanout.
[positional-arguments]
bench-ts-schedule *args:
Expand Down
2 changes: 1 addition & 1 deletion rs/moq-audio/src/encode/producer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -849,7 +849,7 @@ mod tests {
}

#[tokio::test]
#[ignore = "measures real codec CPU; run with just rs bench-playback"]
#[ignore = "measures real codec CPU; release benchmark, run by nightly"]
async fn benchmark_raw_audio() {
println!("tracks,retention_secs,always_on,idle_ms,catchup_ms");
for tracks in [1, 8] {
Expand Down
2 changes: 1 addition & 1 deletion rs/moq-audio/src/playback/mixer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -550,7 +550,7 @@ mod tests {

/// Measurement-only harness: no device and no timing acceptance threshold.
#[test]
#[ignore = "run with just rs bench-playback"]
#[ignore = "release benchmark, run by nightly"]
#[expect(
clippy::disallowed_methods,
clippy::disallowed_types,
Expand Down
2 changes: 1 addition & 1 deletion rs/moq-audio/src/resample.rs
Original file line number Diff line number Diff line change
Expand Up @@ -552,7 +552,7 @@ mod tests {

/// Measurement-only: sweep caller packetization and layout without timing gates.
#[test]
#[ignore = "run with just rs bench-playback"]
#[ignore = "release benchmark, run by nightly"]
fn benchmark_resampler() {
println!("input_rate,output_rate,channels,chunk,packet,ns_per_input_frame");
for (input_rate, output_rate) in [(44_100, 48_000), (48_000, 44_100), (96_000, 48_000)] {
Expand Down
2 changes: 1 addition & 1 deletion rs/moq-bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -295,7 +295,7 @@ to the results.

### Observation cost

`just rs bench-latency` measures latency recording with 1, 4 and 16 concurrent
`just rs bench-tests -p moq-bench --run-ignored only -E 'test(benchmark_observation_contention)'` measures latency recording with 1, 4 and 16 concurrent
writers and 0, 10 and 100 snapshots per run. It checks final sample conservation
and prints the cost per sample for each case. The nightly workflow runs the same
matrix and records it on the benchmark table.
54 changes: 27 additions & 27 deletions sh/gh/nightly.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# Keep benchmark builds together while unrelated Nightly workloads run independently.
set -euo pipefail

usage='usage: just gh nightly rust-test|rust-bench|rust-shared|rust-relay|rust-extra|rust-uring|js|bindings'
usage='usage: just gh nightly rust-test|rust-test-apps|rust-bench|rust-shared|rust-relay|rust-extra|rust-uring|js|bindings'
if [[ $# != 1 ]]; then
echo "$usage" >&2
exit 2
Expand All @@ -12,7 +12,7 @@ fi
case "$1" in
bindings) report= ;;
rust-bench | rust-shared | rust-relay | rust-extra | rust-uring) report=rust ;;
rust-test) report=rust-test ;;
rust-test | rust-test-apps) report=rust-test ;;
js) report=js ;;
*)
echo "$usage" >&2
Expand Down Expand Up @@ -73,43 +73,43 @@ javascript() {
# The bench nextest profile stores each test's printed tables in its JUnit report,
# which every run overwrites; keep each run's copy.
nextest() {
local junit=${CARGO_TARGET_DIR:-$root/target}/nextest/bench/junit.xml
local junit=${CARGO_TARGET_DIR:-$root/target}/nextest/bench/junit.xml report name
rm -f "$junit"
run "$@"
if [[ -s $junit ]]; then
mv "$junit" "$out/raw/nextest-$((++nextest_runs)).xml"
else
if [[ ! -s $junit ]]; then
failed+=("no JUnit report from: $*")
return
fi
report=$out/raw/nextest-$((++nextest_runs)).xml
mv "$junit" "$report"
# A filter that matches nothing passes, as when its test needs a feature the build lacks.
while read -r name; do
if ! grep -qE "<testcase name=\"([^\"]*::)?$name(::[^\"]*)?\"" "$report"; then
failed+=("no test ran for $name in: $*")
fi
done < <(grep -oE 'test\(=?[[:alnum:]_:]+\)' <<<"$*" | sed -E 's/^test\(=?//; s/\)$//')
}
nextest_runs=0

case "$1" in
# Ignored benchmark tests under nextest, all in release builds and one at a time, so
# none measures beside another. Each prints CSV tables whose unit-suffixed columns
# the report records (see bench/README.md).
# the report records (see bench/README.md). A release build links every test binary
# with fat LTO, so each group is one build of only the targets holding its benchmarks,
# and the transport stack and the crates above it split across jobs.
rust-test)
export NEXTEST_PROFILE=bench NEXTEST_TEST_THREADS=1
nextest just rs bench-latency
nextest just rs test -p moq-bench --release --run-ignored only \
-E 'test(benchmark_readiness_establishment)' --success-output immediate
nextest just rs bench-playback
nextest just rs test --release -p moq-congestion -E 'test(benchmark_per_packet)' \
--run-ignored only --success-output immediate
nextest just rs bench-feed-fetch
nextest just rs test -p moq-quic --release -E 'test(benchmark_receive_credit)' --run-ignored only --success-output immediate
nextest just rs test -p moq-tokio --release -E 'test(benchmark_receive_windows)' --run-ignored only --success-output immediate
nextest just rs test -p moq-uring --release -E 'test(benchmark_socket_queues)' --run-ignored only --success-output immediate
nextest just rs test -p moq-tokio --release -E 'test(benchmark_datagram_delivery)' --run-ignored only --success-output immediate
nextest just rs test -p moq-net --release --features async --test integration -E 'test(=mesh_sweep::benchmark_route_mesh)' --run-ignored only --success-output immediate
nextest just rs test -p moq-net --release --lib -E 'test(benchmark_route_rules) | test(benchmark_route_churn)' --run-ignored only --success-output immediate
# One build for the crates whose benchmarks need no features of their own.
nextest just rs test --release --no-fail-fast -p moq-relay -p moq-c -p moq-mux -p moq-cli \
-E 'test(benchmark_stun_sources) | test(benchmark_handles) | test(benchmark_damage_pid_fanout) | test(benchmark_pes_staging) | test(benchmark_stream_import)' \
--run-ignored only --success-output immediate
# Codecs are irrelevant to timebase reservation, so skip building them.
nextest just rs test -p moq-ffi --release --no-default-features -E 'test(benchmark_timebase_reservation_fanout)' \
--run-ignored only --success-output immediate
nextest just rs bench-tests --no-fail-fast --run-ignored only \
-p moq-bench -p moq-congestion -p moq-quic -p moq-net -p moq-tokio -p moq-uring \
--features moq-net/async --lib --bin moq-bench --test integration \
-E 'test(benchmark_observation_contention) | test(benchmark_readiness_establishment) | test(benchmark_per_packet) | test(benchmark_feed_fetch_wakes) | test(=mesh_sweep::benchmark_route_mesh) | test(benchmark_route_rules) | test(benchmark_route_churn) | test(benchmark_receive_credit) | test(benchmark_receive_windows) | test(benchmark_datagram_delivery) | test(benchmark_socket_queues)'
;;
rust-test-apps)
export NEXTEST_PROFILE=bench NEXTEST_TEST_THREADS=1
nextest just rs bench-tests --no-fail-fast --run-ignored only \
-p moq-audio -p moq-mux -p moq-c -p moq-relay -p moq-cli -p moq-ffi \
--features moq-audio/playback --lib --bin moq \
-E 'test(benchmark_mixer_callbacks) | test(benchmark_resampler) | test(benchmark_raw_audio) | test(benchmark_damage_pid_fanout) | test(benchmark_pes_staging) | test(benchmark_handles) | test(benchmark_stun_sources) | test(benchmark_stream_import) | test(benchmark_timebase_reservation_fanout)'
;;
# Measured Cargo bench targets, all in the bench profile. A paired run builds and
# measures each group twice, so the groups split across jobs to fit an hour each.
Expand Down
Loading