Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 51 additions & 6 deletions quest/m1/release-perf/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# [M] Main is no slower than release
# Main is no slower than release

## Goal

Expand All @@ -7,11 +7,11 @@ Before the next release cut, every benchmark where `main` is slower than
is fixed. Cost that pays for a feature `main` deliberately added is accepted,
and the reason is written here.

This epic's own work: once every child has landed, confirm the sparse rows
and the relay fanout load against `release` (see the 2026-10-25 decisions).
The browser frame path needs no repeat; the
[performance audit](/quest/m2/audit-perf.md) now pairs every later cut with
`release`.
This epic's own work: once its child has landed, re-pair the relay `fanout`
workload against `release` and confirm main's relay CPU is no higher in every
round. The sparse rows are accepted (2026-10-30). The browser frame path
needs no repeat; the [performance audit](/quest/m2/audit-perf.md) now pairs
every later cut with `release`.

## Plan

Expand Down Expand Up @@ -227,6 +227,51 @@ Decided 2026-10-25 (maintainer), in
the sparse rows). Fix it if the relay
CPU still rises with consistent sign; otherwise this epic is done.

### Re-run (2026-10-30)

`release` (`de0a2b0534`) against `main` (`7c8e255304`). The host was shared
with other agents: 1-minute load 11 to 29 on 24 cores. Rather than the full
`just bench origin/release`, this run paired only the relay workloads (the
tail of `bench/run.sh`, seven alternating rounds) and `track_sparse_read`
(three alternating rounds). Release's `track` bench has no `async` feature, so
`run.sh --criterion` cannot build both sides with one set of arguments.

Relay fanout is bimodal on `main`. Release's relay used 0.062 to 0.082 cores
in all seven rounds. Main's used 0.062 to 0.072 in three rounds and 0.157 to
0.241 in four, with the same delivery (640 frames/s, p99 within 2%). The
comparator calls the sign mixed (paired deltas -15% to +218%, median +119%),
but the high runs share a mechanism. In them, 21 to 24 Tokio workers are busy
and the relay makes 9,400 to 12,600 voluntary context switches/s, against 2
to 6 busy workers and 400 to 1,300 switches/s otherwise. Release never
exceeded 4 busy workers or 2,800 switches/s. Eight CPU-profiled runs of
`main` reproduced it in five. In a high run, 54% of samples end in a syscall
(3 of 227 in a low run). Most are futex waits on the track's shared
`kio::Lock`, taken by every subscriber's `read` in
`poll_recv_next → poll_peek_group` and
`poll_forward_drop → poll_drop`, plus the wakes that follow. When the 64
subscriber tasks land on many workers, every frame makes them contend that
one lock; packed onto a few workers, they never contend. Video used 9.7% less
CPU (MAD 7.1%), with 1.4% more frames delivered.

`track_sparse_read/timed` is flat now that
[moq-dev/moq#6921](https://github.com/moq-dev/moq/pull/6921) has landed:
release 2.42, 2.49, 2.64 us and main 5.33, 5.98, 6.42 us at 16, 256, 4096
gaps (medians of three rounds). From 16 to 4096 gaps, main grows 20% and
release 9%. The 2.2 to 2.4x gap is per iteration, not per gap. Each iteration
creates, writes, finishes, and receives two groups, so this is the per-group
model cost accepted above.

Decided 2026-10-30 (maintainer), in
[moq-dev/moq#7299](https://github.com/moq-dev/moq/pull/7299):

- The sparse rows are accepted as that per-group model cost.
- The fanout mode is accidental and is fixed before the cut, in
[Fanout lock contention](/quest/m1/release-perf/fanout-lock.md).

## Required

- [Fanout lock contention](/quest/m1/release-perf/fanout-lock.md) - relay fanout subscribers stop contending one track lock, so main's relay CPU matches release's in every round

## Related

- [Performance audit](/quest/m2/audit-perf.md) - compares nightly to nightly; this epic compares main to release
Expand Down
59 changes: 59 additions & 0 deletions quest/m1/release-perf/fanout-lock.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# [M] Relay fanout subscribers stop contending one track lock

## Goal

The relay's `fanout` workload (`bench/relay.sh`, 64 subscribers of one track)
uses no more relay CPU on `main` than on `release` in every alternating
round, with the same delivery. Subscribers of one track no longer serialize
on the track's shared `kio::Lock` for every frame, so spreading their tasks
across Tokio workers no longer multiplies the relay's CPU.

## Plan

Found by the 2026-10-30 re-run in the
[release-perf epic](/quest/m1/release-perf/README.md)
([moq-dev/moq#7299](https://github.com/moq-dev/moq/pull/7299)), `release`
`de0a2b0534` against `main` `7c8e255304`. Decided 2026-10-30 (maintainer):
this is accidental cost, fixed before the cut.

What the profile found:

- `main`'s relay is bimodal under `fanout`. Release used 0.062 to 0.082
cores in all seven rounds; main used 0.062 to 0.072 in three and 0.157 to
0.241 in four, at the same 640 frames/s and p99.
- In the high mode 21 to 24 Tokio workers are busy and the relay makes 9,400
to 12,600 voluntary context switches/s, against 2 to 6 workers and 400 to
1,300 switches/s in the low mode. Release never exceeded 4 busy workers or
2,800 switches/s. Eight CPU-profiled runs of `main` (frame pointers, the
default shared runtime) reproduced the high mode in five.
- In a high run 54% of samples end in a syscall (3 of 227 in a low run),
mostly futex waits on the track's shared `kio::Lock` and the wakes after
them. Every subscriber's read takes it, in the lite publisher's
`poll_recv_next` → `poll_group` / `poll_peek_group` and `poll_forward_drop`
→ `poll_drop` (`rs/moq-net/src/lite/publisher.rs`), which reach the track
state through `rs/moq-net/src/model/resume.rs` and
`rs/moq-net/src/model/track.rs`.
- Packed onto a few workers, the 64 tasks never contend. Spread over many,
each frame wakes them all onto one mutex.

Approach:

- First find what `release`'s read path did differently, since release ran in
the same rounds and never spread. Compare how many times one subscriber's
wake takes the track lock on each branch.
- Then cut the per-wake lock work: take the lock once per wake rather than
once per poll step, or keep per-reader state outside the producer's mutex
so a reader only locks when a new group or drop is published.
- Add a Criterion case that reproduces the contention without the relay
harness (many readers of one track on a multi-threaded runtime), and keep
it in the release pairing.
- Verify with seven alternating `fanout` rounds against `origin/release`,
recording `/proc/loadavg`, busy workers, and voluntary context switches
per run; a quiet host is preferred. `just bench-profile` cannot yet run the
default shared runtime, where the mode shows up.

Public API: none expected. Wire: none.

## Related

- [Log lock profile](/quest/m1/unbounded-groups/log-lock-profile.md) - lock contention profiling of `GroupState`, not the track lock
Loading