[kixelated]

Problem

Under the relay fanout workload (64 subscribers of one track), main's relay CPU was bimodal. In some rounds the 64 subscription tasks spread over every Tokio worker and serialized on the track's shared kio::Lock, and relay CPU went to 0.18 to 0.23 cores with about 10k context switches/s. Release stayed at about 0.08. See the release-perf epic and moq-dev/moq#7299.

Finishes quest/m1/release-perf/fanout-lock.md, which this PR deletes. The epic keeps its own remaining work: a re-pair on a quieter host once this lands.

What release did differently

I counted every kio::Lock::lock by type and caller, using an instrumented build on one worker under the real fanout workload. Per delivered group:

lock	main	release	this PR
the track's TrackState (shared by all 64 readers)	155	31	34
the logical track's route (Route / release's ResumeState)	53	27	41
every kio lock	365	238	217

Both branches poll each subscription about 4 times per group. On main, each of those polls walked the logical-track reader (resume::Reader) several times (poll_drop twice, poll_recv_group, datagrams, poll_finished), and each walk re-took the track lock:

• Reader::learn_end(None) ran about 10 times per poll (from sync, let_go, and poll_finished), and each run polled every copy's end under the lock. That was 33% of the futex wait in a high-mode profile.
• Cursor::probe took the lock 3 times: once to clone the fills, once to judge, and once to expire. That was 17%.
• let_go read the serving copy's latest_sequence (2 locks) even when no replaced copy needed it. That was 12%.
• Cursor::sync took the lock with a settled window and nothing to read. That was 13%.
• Every group machine's frame wake re-ran the whole track side.

The high-mode profile came from perf context-switch stacks through lock_contended. 98% of the futex wait was on TrackState.

Approach

The Plan offered two directions: take the lock once per wake, or keep per-reader state outside the producer's mutex. This PR does both, each as a narrow change:

• The lite publisher skips its track side when nothing on it fired (rs/moq-net/src/lite/publisher.rs, new rs/moq-net/src/util/idle.rs). TrackRun::poll_step is split. The prelude (flush, stream credit, group machines, drops) runs on every wake. The track side (poll_track) parks on a waiter of its own, which records any wake before it wakes the task. While that waiter hasn't fired, a repeated poll returns Pending without touching the track. Three things change the track side's answer without a notification, so each disarms the gate: a SUBSCRIBE_UPDATE, the group waiting for credit getting a stream, and a group machine retiring.
• Learning the end reads a lock-free flag first (cache::Track::end_declared, track::Subscriber::declared_end). TrackState::set_end sets it under the lock, and nothing clears it. Until a track declares an end, learn_end(None) skips its lock. Once it has, the reader takes the same locked read as before.
• Locks only taken when needed (model/resume.rs, model/track.rs, model/track/miss.rs). let_go reads the edge only when a copy is replaced, which is the only case that reads it. Cursor::sync returns early for a settled window with no new ranges. probe judges and clones the fills under one lock. It skips expire when no fill owes anything, and next_filled skips the lookup when nothing is filled. Each of these returns exactly what the locked read returned before.
• Bench: session_fanout_threads serves one broadcast's one-frame groups through one relay to 64 viewers, on 1 and 16 Tokio workers. It is timed in process CPU, which adds Clock::Process beside the existing thread clock. It is skipped off Unix. moq-net's dev dependency on tokio gains rt-multi-thread.

Public API: none. Wire: none.

Measurements

Three setups:

• Pinned relay: one worker, pinned to P-core 2. Instructions are perf stat -e instructions:u over a 5 s window (3,200 deliveries).
• Default runtime: 24 workers, alternating rounds. The relays are built with the profiling profile, plus frame pointers.
• Bench: criterion, process CPU per round of 64 groups.

All runs used the same load generator, built from main. The host is a shared 24-core machine at load 7 to 28. I had no quiet host: nixos2 stopped answering.

Pinned relay, user-space instructions per 5 s

main	release	this PR
472M to 527M	401M to 512M	394M to 409M

Default runtime, 7 alternating rounds (cores; vol. switches/s; busy workers; instructions)

round	release	this PR	main
1	0.082; 1766; 2; 1.10G	0.068; 734; 5; 0.89G	0.067; 415; 5; 1.06G
2	0.073; 490; 2; 0.91G	0.071; 866; 0; 0.59G	0.201; 9978; 24; 1.13G
3	0.078; 1824; 3; 1.04G	0.084; 2091; 1; 0.88G	0.083; 1360; 7; 1.11G
4	0.080; 1375; 2; 0.96G	0.076; 1435; 2; 0.87G	0.177; 9358; 23; 1.16G
5	0.080; 1813; 1; 1.06G	0.074; 1244; 2; 0.89G	0.120; 4641; 9; 1.11G
6	0.076; 1155; 2; 1.01G	0.071; 493; 4; 0.82G	0.072; 447; 6; 1.08G
7	0.072; 429; 4; 1.09G	0.082; 1693; 0; 0.87G	0.233; 12648; 24; 1.19G

Delivery was the same in every run: 640 frames/s, p99 within the run-to-run spread. A second set of 7 rounds of release against this PR gave 0.062 to 0.104 cores for release and 0.066 to 0.090 for this PR.

Over the 14 rounds, this PR never entered the high mode: at most 5 busy workers and 2.1k switches/s. Its paired CPU deltas against release have mixed sign, 4 of 14 above release, all within release's own round-to-round spread. It ran 5 to 36% fewer user-space instructions than release in every round. main went high in 4 of 7.

session_fanout_threads, process CPU per round

case	main	this PR
moq-lite-06/workers=1	3.47 ms	1.47 ms
moq-lite-07-wip/workers=1	3.95 ms	1.83 ms
moq-lite-06/workers=16	22.6 ms	4.19 ms
moq-lite-07-wip/workers=16	26.9 ms	7.07 ms

Validation

• nix develop -c quest gate just ci origin/main passed at e36eb37755: 9,263 tests passed, 101 skipped, and every check module passed.
• After the latest base merge, just rs test passed for moq-net, moq-net-tests, and moq-relay: 3,661 tests.
• cargo clippy -p moq-net --all-targets --all-features and --no-default-features --all-targets are clean with -D warnings.
• Mutation checks on the gate:
  • A gate that never reopens hangs moq-net's lib tests: 8 still running after 60 s.
  • Dropping the disarm when the credit-waiting group gets a stream fails credit_after_a_stall_serves_the_live_edge.
  • Dropping the update or retired-machine disarm fails nothing, so something the track side parked on also fires on those paths today. I kept both disarms because the track side's answer reads the state they guard, and I didn't want correctness to depend on another module's notification.
• New unit tests in util/idle.rs: a parked read stays parked until it is woken or disarmed, or until another task reads. A waiter parked past its records is retired.

Choices made without asking

• Both Plan directions. I implemented both, as listed under Approach, rather than restructuring the reader to lock once per wake. The counts showed each redundant acquisition had a local fix that returns exactly what the locked read did. The gate cuts the remaining whole-walk repeats, from 4.06 to 3.04 track walks per delivery.
• Gate mechanism. util::Idle is the same mechanism as the cursor gate from moq-dev/moq#7263, which landed first. That one gates Cursor group reads on a plain track. This one gates the publisher's whole track side, which reaches a relay's logical track through resume::Reader, where perf(moq-net): an idle track read parks again without walking the read path #7263's gate doesn't apply. Since this PR landed second, the two now share one type: util::Idle<K> is keyed by the read that parked, the cursor's gate is util::Idle<Walk> (model/track/idle.rs keeps only Walk), and the publisher's is util::Idle<()>.
• Bench workers. session_fanout_threads sweeps 1 and 16 workers. 16 shows the contention on a 24-core host without tying the bench to the host's core count.

Decisions

1. Keep session_fanout_threads in the release pairing? The Plan says to keep the new Criterion case in the release pairing, but release doesn't have it, and a pairing compares only cases both sides have.
   • ✅ A: open a bench-only backport to release, so the pre-cut pairing compares it. Decided 2026-11-01 (maintainer): "I'm fine with bench only backports going forward, check what other benchmarks could be backported." The other backportable benches are in moq-dev/moq#7586; this case follows it once this PR lands, ported into release's single-file session bench. The rule change that allows bench-only backports is moq-dev/moq#7581.
   • B: don't backport.

Follow-ups

• Route lock: 41 acquisitions per delivery remain. Reader::sync reads the route state on every call, and Route::end and poll_changed are called on each walk. An atomic copy of the route generation would let an unchanged route skip it.
• GroupExpiry::is_expired takes the track lock on each group-machine poll once a newer group has landed. That was 5.7% of the futex wait in main's profile.
• bench/profile.py lock accepts only moq-uring-N and moq-quic-N workers. Accepting tokio-rt-worker would let just bench-profile run the default shared runtime; the quest noted it could not.
• setup::tests::credentials_stay_out_of_traces failed twice under plain cargo test --lib. It passes alone and under nextest, and the failure looks unrelated to this change. I didn't check it on main.

Review

Codex gpt-6.1-sol, medium effort, via quest review --base origin/main.

• Round 1, head e36eb37755: 1 finding, fixed in 5211f4fbc4. [medium] session_fanout_threads selected Clock::Process, which panics off Unix. It now returns first when !cfg!(unix).
• Round 2, head 5211f4fbc4: 0 findings.
• New session, round 1, head feafb43d8c: 0 findings. This head merges origin/main (bf4e726f66), resolving a conflict in Cursor::probe against moq-dev/moq#7263's lazy Misses::retain: the probe keeps main's unconditional retain call and this PR's single lock for the judgement and the fills. It also shares one util::Idle between the two gates, records the decision above in the epic, and points relay-cpu-bisect.md at this PR instead of the deleted quest.

Checks at feafb43d8c: cargo clippy -p moq-net --all-targets with --all-features and with --no-default-features, -D warnings, clean. cargo nextest run -p moq-net --all-features: 3,223 passed. cargo nextest run -p moq-net-tests: 39 passed. quest check --base origin/main: ok.

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 36 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 9386d100-661d-4cda-aa83-ca87ce4d8ff8

Commits

Reviewing files that changed from the base of the PR and between bf4e726 and feafb43.

Files selected for processing: 14

• quest/m1/release-perf/README.md
• quest/m1/release-perf/fanout-lock.md
• quest/m1/release-perf/relay-cpu-bisect.md
• rs/moq-net/Cargo.toml
• rs/moq-net/benches/session.rs
• rs/moq-net/benches/session/mod.rs
• rs/moq-net/src/lite/publisher.rs
• rs/moq-net/src/model/cache.rs
• rs/moq-net/src/model/resume.rs
• rs/moq-net/src/model/track.rs
• rs/moq-net/src/model/track/idle.rs
• rs/moq-net/src/model/track/miss.rs
• rs/moq-net/src/util.rs
• rs/moq-net/src/util/idle.rs

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

[kixelated]

Automated review by review (OpenAI)

Reviewed commit: feafb43

No actionable bugs found in the changed paths. The direction is sound: eliminate redundant shared-track reads, and reuse the existing keyed idle gate rather than introduce a second wake-tracking mechanism. The publisher keeps flushing and driving group machines outside the gate; updates, stream-credit progress and machine retirement explicitly reopen it. The end flag still defers reading the actual end to the track lock. No public API or wire-format changes identified.

Verification: static inspection of the diff and relevant wake, cursor, end-learning and benchmark code only. I did not run tests or benchmarks or independently reproduce the reported performance results.

(Written by OpenAI)

[pullfrog]

✅ No new issues found.

Reviewed changes

• Lite publisher idle gate: TrackRun::poll_step is split so the track side (poll_track) is skipped while its Idle gate is parked, disarmed on SUBSCRIBE_UPDATE, a resolved opening group, and any child-machine retirement. Every poll_track Pending return is backed by a registration on the gate's waiter or one of those disarms.
• Lock-free end flag: cache::Track records that an end was declared, so Subscriber::declared_end (used by resume::Reader::learn_end) skips the track lock until there is an end to read, and otherwise returns exactly the previous poll_finished result.
• Cheaper locked reads: let_go computes the retention edge only when a copy is replaced, Cursor::sync returns early for a settled window, probe judges and clones fills under one lock, and expire/next_filled are skipped when they owe nothing, each matching the removed code's own early-outs.
• Shared util::Idle<K>: the cursor gate from #7263 is generalized and reused by the publisher with K = (); park/wake semantics are unchanged and the keyed-read test is added.
• Bench + quests: session_fanout_threads reproduces the fanout contention on 1/16 workers in process CPU (Unix-only, skipped under the alloc/expiry modes); the finished quest is deleted and the epic plus bisect note are repointed with no dangling links.

I verified cargo check -p moq-net --all-targets --all-features and --no-default-features are clean (including the new bench), and cargo test -p moq-net --lib util::idle passes.

  ^{｜ View workflow run ｜ Using deepseek-v4.1-flash (free via Pullfrog for OSS) ｜ 𝕏}