Skip to content

test(hls): record and serve a broadcast across a moq-net session - #6776

Merged
kixelated merged 8 commits into
mainfrom
quest/m1/hls-session-tests
Oct 25, 2026
Merged

kixelated merged 8 commits into
mainfrom
quest/m1/hls-session-tests

Conversation

@kixelated

@kixelated kixelated commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

Completes quest/m1/hls-session-tests.md and splits the takeover scenario into the new quest/m1/hls-takeover-leak.md.

Summary

A new rs/moq-hls/tests/integration suite runs a publisher and the HLS export on either side of an in-memory moq-net session (transport::memory::pair, Client/Server) on the paused clock, through moq-hls's public API only. The export is the client, as it is when it connects to a relay.

  • Disconnect (disconnect.rs): the publisher's session drops mid-segment (a GOP open) or right after a catalog update. The cursor drains the completed tail, then fails every call with CatalogFailed, TimelineFailed, or SegmentUnavailable. The media playlist refuses, and HTTP does not answer 200.
  • Clean finish (finish.rs): the cursor reads to None and the playlist ends with EXT-X-ENDLIST, with announce End arriving both after and before catalog EOF.
  • Server (server.rs): the 7 server/routes.rs tests that crossed a session over loopback TCP on the wall clock now run on the shared harness. LitePair and its listed helper are deleted, as the quest's "promote, don't duplicate" asks. The local-origin routes tests stay inline.
  • Takeover: moved to quest/m1/hls-takeover-leak.md, which holds the scenario inline. On main, after a new instance takes over, the replaced run's recording cursor yields its final segment with the new run's media, or stalls if the new run has not completed that group. This reproduces on moq-lite-06 and moq-lite-07-wip. The quest records the evidence so far: the fetch goes through the old instance's held track query yet returns the replacement's group, and a bare moq-net reproduction did not leak. It also lists the code already examined and a leading hypothesis. The quest lands the scenario together with the fix.

Regression check

The moq-lite session tears subscriptions down in HashMap order (SubscriptionCleanup in lite/subscriber.rs), so whether the catalog or the timeline failure reaches the cursor first changes from run to run. Each disconnect test therefore repeats the scenario on fresh sessions until it has seen both orders, capped at 64 runs, and asserts that every run ends typed. With #6694's CatalogFailed wrapping reverted, both disconnect tests failed in 15 of 15 runs. With the wrapping in place they passed in 10 of 10.

Public API and wire impact

None. Only tests and quests change.

Choices made

  • Default moq-lite version only for the new scenarios. The two ported tests that loop over versions keep their loops.
  • The harness reads each instance's URLs from the master playlist instead of the crate-private Broadcaster::instance.
  • The ported tests now run on the paused clock over the memory transport instead of TCP on the wall clock.
  • The takeover scenario was split into its own quest, per the maintainer's pick, so the rest lands now.

Checks

  • just check: passes, 478 tests. One test was reported SLOW: moq-cli::integration report::export_echo_reports_where_the_import_solicits. moq-cli is selected only because it depends on moq-hls; this PR does not touch that test.
  • quest check --base origin/main: ok.
  • Not run: just test interop (no wire change).

Follow-ups

  • quest/m1/hls-takeover-leak.md: the leak, plus the scenario that guards it.
  • moq-hls's moq-tokio dev-dependency features (tcp, quic, aws-lc-rs) are now used only for origin::spawn. Trimming them to net first needs a fix in moq-tokio: Backoff::validate is dead code under net alone. This is noted in the new quest's Related section.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

kixelated and others added 2 commits October 25, 2026 02:18
Adds rs/moq-hls/tests/integration: a publisher and the export on either
side of an in-memory moq-net session on the paused clock, through the
public API only. It covers a publisher disconnect (both failure orders),
a clean finish with End before and after catalog EOF, and a takeover.
The routes.rs tests that crossed a session over loopback TCP move into
the suite, and LitePair goes with them.

The takeover scenario fails on main: the replaced run's cursor yields
its final segment with the new run's media.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title HLS export tests over a session test(hls): record and serve a broadcast across a moq-net session Oct 25, 2026
The takeover scenario fails on main: a replaced run's cursor yields the
replacement's media. It moves, with what is known about the leak, into
quest/m1/hls-takeover-leak.md, which lands it with the fix. The session
suite's quest is complete and removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review October 25, 2026 09:47
@coderabbitai

coderabbitai Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 32 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: cf08da1a-4fd2-4ea0-8549-f3664d6cd446
Commits

Reviewing files that changed from the base of the PR and between bd3c597 and a47616a.

Files selected for processing: 9
  • quest/m1/README.md
  • quest/m1/hls-session-tests.md
  • quest/m1/hls-takeover-leak.md
  • rs/moq-hls/src/server/routes.rs
  • rs/moq-hls/tests/integration/disconnect.rs
  • rs/moq-hls/tests/integration/finish.rs
  • rs/moq-hls/tests/integration/main.rs
  • rs/moq-hls/tests/integration/server.rs
  • rs/moq-hls/tests/integration/support.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-25T20:16:39.281560Z a47616a New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: 3fd54eb

  • P2, rs/moq-hls/tests/integration/disconnect.rs:110–115: Force the two failure orders rather than requiring both within 64 repetitions. Teardown order is unspecified, and each run accepts either typed failure, yet the suite fails if a valid ordering consistently wins. Random HashMap iteration and executor scheduling do not guarantee both outcomes in this bound. Add explicit ordering control so both paths are exercised deterministically.

Direction: session coverage behind the public API with an in-memory transport is useful. The existing takeover leak is explicitly tracked separately, not introduced by these test-only changes.

Verification: static review of the tests and surrounding session/teardown code. Tests were not run.

# Conflicts:
#	quest/m1/README.md
#	rs/moq-hls/src/server/routes.rs
@kixelated

Copy link
Copy Markdown
Collaborator Author

Merged main (b032863). The moved session tests and the shared Publisher fixture now build through origin::Publication, matching main's edits to the routes.rs copies this PR deletes. Main changed nothing else in those tests. The m1 README conflict keeps only the new takeover-leak quest.

On the OpenAI P2 (force both disconnect orders instead of requiring both within 64 runs), I disagree, so the test stays as is:

  • I measured the split over 6 batches of 40 runs on this head. The catalog failure landed first 50 to 70% of the time, the timeline failure the rest, and the segment outcome never did. At the worst observed split, 64 runs miss one order with probability about 1e-13.
  • The order comes from HashMap iteration in moq-lite's subscription teardown. Forcing it would need a teardown-order hook in moq-net's session, which is production code added only for a test.
  • If a later change makes the order deterministic, the cap fails loudly ("pin the other order another way"). That is intended: otherwise one of the two typed paths would silently lose coverage.

Each run is otherwise deterministic on the paused clock. cargo test -p moq-hls --all-features and quest check --base origin/main pass.

(Written by Claude Opus 5.5)

@kixelated
kixelated added this pull request to the merge queue Oct 25, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b03286303f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 25, 2026
kixelated and others added 2 commits October 25, 2026 11:03
Since #6741 a closed lite session drains its sources in a fixed order, so
the catalog failure always reaches the cursor first and the loop waiting
for both orders could never finish. Each disconnect now runs once and
accepts any typed failure; export unit tests cover the timeline failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated added this pull request to the merge queue Oct 25, 2026
@kixelated

Copy link
Copy Markdown
Collaborator Author

Left the merge queue on failed checks: since #6741 a closed lite session drains its sources in a fixed order, so disconnect::mid_* saw the catalog failure land first 64 times and never the timeline one. Merged main and fixed the test in 5ae5972: each disconnect runs once and must end with a typed failure, whichever subscription lands first. Re-queued after a Codex thumbs-up on 5ae5972.

(Written by Claude Opus 5.5)

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: 5ae5972

Fixed: the previous teardown-order finding #6776 (review). Each disconnect now checks the recording contract once instead of requiring both unspecified outcomes within 64 runs.

No new actionable bugs found. Direction: this is simpler and avoids coupling session tests to teardown order; dedicated export tests retain explicit timeline-failure coverage.

Verification: static comparison against 3fd54eb and relevant tests, accounting for merged base changes. Tests were not run.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Oct 25, 2026
kixelated and others added 2 commits October 25, 2026 13:14
# Conflicts:
#	quest/m1/README.md
#	quest/m1/hls-session-tests.md
#	rs/moq-hls/src/server/routes.rs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated added this pull request to the merge queue Oct 25, 2026
@kixelated

Copy link
Copy Markdown
Collaborator Author

Left the merge queue on a conflict after #6777 landed. Merged main: kept this PR's removal of the moved session tests from routes.rs, and ported 6777's segment.media Option change to the moved copy in tests/integration/server.rs. Also repointed hls-takeover-leak.md's link from the finished recorder quest to its code. Re-queued.

(Written by Claude Opus 5.5)

Merged via the queue into main with commit 4e50d3b Oct 25, 2026
10 checks passed
@kixelated
kixelated deleted the quest/m1/hls-session-tests branch October 25, 2026 21:33
kixelated added a commit that referenced this pull request Oct 25, 2026
Ports main's changes onto per-rendition HLS records:
- #6757: `retire` moves to `hls::Tracks::retire`, and the TS PMT churn test counts record tracks.
- #6777 and #6776: the recorder and session tests publish with `with_hls`, read stops as gaps, and take segment tags from the rendition.
- The segment cursor yields gaps, as main now does.

A reconfigured rendition's segment tag now folds in its normalized config, and its replacement lists only the records after its predecessor's, so one URL never names two muxes. A dropping pacer no longer makes followers stop early.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant