[kixelated]

Re-runs the release pairing the 2026-10-25 decision asked for (relay fanout plus the sparse rows), now that moq-dev/moq#6921 has landed, and records the result in quest/m1/release-perf/README.md. The epic has no Required children; this is its own remaining work.

Measurements

release (de0a2b0534) against main (7c8e255304). The host was shared with other agents' builds: 1-minute load 11 to 29 on 24 cores. A quiet host was not available.

Relay fanout is bimodal on main. Seven alternating rounds:

	relay CPU (cores) per round
release	0.078 0.082 0.074 0.076 0.068 0.072 0.062
main	0.241 0.069 0.166 0.072 0.062 0.157 0.197

Delivery was the same on both (640 frames/s, p99 within 2%). compare.py reports "sign varies across pairs": paired deltas from −15% to +218%, median +119%, MAD 99%. The high runs share one mechanism:

• 21 to 24 Tokio workers are busy, with 9.4k to 12.6k voluntary context switches/s. The low runs have 2 to 6 busy workers and 0.4k to 1.3k switches/s. Release never exceeded 4 busy workers or 2.8k switches/s.
• Eight CPU-profiled runs of main (frame pointers, default shared runtime) reproduced the high mode in five.
• In a high run, 54% of samples end in a syscall, against 3 of 227 in a low run. Most are futex waits on the track's shared kio::Lock, taken by every subscriber's read in poll_recv_next → poll_group → poll_peek_group and poll_forward_drop → poll_drop, plus the wakes that follow.
• When the 64 subscriber tasks are spread over many workers, every frame makes them contend that one lock. Packed onto a few workers, they never contend.

Video used 9.7% less relay CPU (MAD 7.1%), with 1.4% more frames delivered.

track_sparse_read/timed is flat now. Medians of three alternating rounds:

gaps	release	main	ratio
16	2.42 µs	5.33 µs	2.20x
256	2.49 µs	5.98 µs	2.40x
4096	2.64 µs	6.42 µs	2.43x

From 16 to 4096 gaps, main grows 20% and release 9% (it was 4.3x at 4096 before moq-dev/moq#6921). Both benches do the same work per iteration: create, write, finish, and receive two groups. The remaining gap is the per-group model cost the README already accepts.

Decisions

1. ✅ Relay fanout: fix it before the cut (maintainer, 2026-10-30). This PR adds the child quest quest/m1/release-perf/fanout-lock.md, ranked in the epic's Required. It carries what the profile found (the bimodal relay CPU, the busy-worker and context-switch counts, and the futex waits on the track's shared kio::Lock under poll_recv_next and poll_forward_drop), an approach, and the re-pair that verifies it.
2. ✅ Sparse rows: accepted (maintainer, 2026-10-30) as the per-group model cost, recorded in the README.

The epic's own work is now the fanout re-pair against release once the child lands. Since it has a child, the README's title drops its size.

Choices made without asking

• I paired only the relay workloads and track_sparse_read, not the full just bench origin/release. A full run is about 1,000 cases on each side plus two full bench builds, over two hours on this host. The Plan's remaining questions were only fanout and sparse.
• I ran those through scratch drivers, not bench/run.sh. The relay driver reuses bench/relay.sh and bench/compare.py unchanged, with the same alternating order. It was needed because run.sh has no relay-only mode. The sparse driver was needed because release's track bench has no async feature, so run.sh --criterion cannot build both sides with one set of cargo arguments.
• I used seven relay rounds instead of the default three, to tighten the fanout spread.

Follow-ups

• bench/run.sh: a relay-only pairing mode, and per-side cargo features in --criterion pairing so it works against release.
• bench/compare.py: flag a bimodal side (one side's spread far above the other's) instead of only "sign varies".
• just bench-profile: allow the default shared runtime, where the fanout mode shows up.

Checks

• quest check --base origin/main: passed on 90b93fd9cb.
• just ci origin/main: passed on 2ba27b6848. 90b93fd9cb only edits quests.

Review

• Reviewer: Codex gpt-6.1-sol, medium effort, via quest review --base origin/main.
• Head: 90b93fd9cb1e27acd106f1c6f900c39a539366ea (round 1, session 01a18e75-4d1d-73f0-be39-294e83cee04c): no findings.
• Earlier head 2ba27b68480653c9c273a316f5c9c49a662763b1: round 1, no findings.

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 7 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: df2f4ef5-6914-4bc1-8605-164bcf381592

Commits

Reviewing files that changed from the base of the PR and between 7c8e255 and 90b93fd.

Files selected for processing: 2

• quest/m1/release-perf/README.md
• quest/m1/release-perf/fanout-lock.md

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-30T20:18:42.249971Z	90b93fd	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[kixelated]

Automated review by review (OpenAI)

Reviewed commit: 90b93fd9cb1e27acd106f1c6f900c39a539366ea.

No actionable bugs found in this documentation/design change. Separating the accepted sparse per-group cost from the required fanout contention fix is a sound direction. Comparing release's read path first, then reducing redundant locking and adding a multi-reader reproducer, keeps the proposed work focused rather than committing prematurely to a broader concurrency redesign.

Verification: statically reviewed both changed quest documents and the referenced publisher/resume/track polling paths and sparse benchmark. I did not run tests, benchmarks, or profiles, and cannot independently verify the reported measurements or causal attribution. No prior published review findings to carry forward.