Skip to content

test(interop): keep the Rust publisher's media as long as the audio readers wait - #6658

Merged
kixelated merged 5 commits into
mainfrom
quest/m1/test-flakes-2/interop-timestamp-gap
Oct 24, 2026
Merged

kixelated merged 5 commits into
mainfrom
quest/m1/test-flakes-2/interop-timestamp-gap

Conversation

@kixelated

@kixelated kixelated commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Quest: quest/m1/test-flakes-2/interop-timestamp-gap.md (deleted here).

Summary

rust -> js-native-node and rust-swapped -> js-native-node intermittently failed the audio check with a timestamp gap up to the next 1 s group (seen again on 2026-10-24 in run 40094095832, where the Opus or AAC rendition jumped from about 200 ms to 1024 ms).

Cause. The Rust publisher imported with --max-age 1s. A track's max-age caps every subscriber's max delay, both at the relay and in the subscriber's own @moq/net track, so the native JS audio reader's 5 s budget (AUDIO_MAX_DELAY, added in moq-dev/moq#6284 for exactly this startup lag) was really 1 s. ffmpeg writes 1 s fMP4 fragments, so each audio group lands as one burst. A reader joining with one group of backlog is served the oldest group while it sits about 980 ms behind the live edge; as soon as the next group lands it is 1 s behind, goes stale, and is cut mid-read. The Container consumer reports a gap, the subscriber moves to the next group, and tone.py fails the discontinuity.

This was the Node subscriber's own track layer and the relay dropping the group correctly by the 1 s budget the publisher declared, not a delivery bug. In the reproduced failures the relay finished every group (finished group sequence=0); the reader's local track expired group 0 when group 2 landed about 40 ms after group 0 started serving. Only the Rust publisher sets a max-age (the FFI publishers keep hang's 30 s), which is why python/go/cpp -> js-native-node never failed.

Fix. The fMP4 publisher keeps its media 5 s (--max-age 5s), covering the longest reader budget: the native JS audio reader and the wire-compat JS reader both ask for 5 s. The comment at AUDIO_MAX_DELAY now says the publisher's max-age caps it. Since moq-dev/moq#6636, --max-age no longer extends how long a finished import of the current CLI stays up, so this costs it no shutdown time. A released CLI in the wire-compat lanes still stays up for its max-age, now 5 s, within stop_publisher's 10 s. The comment above the import said a finished import stays up for its max-age, which that change made stale; it now names the linger.

Reproduction

A loop outside the checkout: the relay, the Rust CLI fed by the same ffmpeg fMP4 command as interop.sh, and subscribe.ts under Node on moq-lite-07-wip, judged by tone.py, with 72 busy loops on a 24-core machine.

Publisher max-age Failures
1 s (before) 7 of 40 (Opus and AAC, each a gap to the next group)
5 s (after) 0 of 40

The real harness (interop.sh --publishers rust --subscribers js-native-node,js-native-bun) then passed 4 of 4 runs under 48 busy loops.

The broadcast discovery failed ... connection error: closed line in the failing log is unrelated: it prints on every native JS run, passing ones included, as the session closes.

Public API and wire impact

None. Test harness only.

Checks

  • Repro loop under load, before and after (above).
  • interop.sh --publishers rust --subscribers js-native-node,js-native-bun, 4 runs under load.
  • just test interop --publishers rust --subscribers js-native-node after merging main.
  • just check passed.

Chosen without asking (please review)

  • Raise the publisher's max-age to 5 s rather than change the reader. The reader already asks for 5 s; the publisher was silently cutting it to the group duration, which leaves a joining reader no slack. Alternatives: start the reader at the newest group (still races at a 1 s budget, and needs a new reader policy), or shorter fMP4 fragments (changes the import under test).
  • The video-only avc3 path keeps --max-age 1s. It has no gap check, and its readers ask for 1 s.

Follow-ups

  • Since moq-dev/moq#6636 the current CLI lingers 30 s after EOF by default, so every Rust matrix round now ends with WARN publisher 'rust' was still running 10 s after it was stopped; killing it (seen in the post-merge run here; the round still passes). The fix is import --linger, which released CLIs in the wire-compat lanes refuse, so it needs a capability check. Not done here: a different problem from the audio gap.
  • The Python, Go, C, and Rust audio readers ask for a 1 s max delay against 1 s groups, so the same race applies to them in principle if one is ever starved at startup. None has failed this way.
  • The native JS subscriber logs broadcast discovery failed ... connection error: closed on every clean exit, which reads like a failure in triage.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

kixelated and others added 5 commits October 24, 2026 10:00
…eaders wait

The Rust CLI imported with --max-age 1s, which caps every reader's max delay,
so the native JS audio reader's 5 s budget was really 1 s. Each 1 s fMP4
fragment lands as one burst, so the oldest group a joining reader is served
went stale as soon as the next group landed and was cut mid-read, which the
tone check failed as a timestamp gap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Interop audio has no timestamp gap under load test(interop): keep the Rust publisher's media as long as the audio readers wait Oct 24, 2026
@kixelated
kixelated marked this pull request as ready for review October 24, 2026 17:46
@coderabbitai

coderabbitai Bot commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 14 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: e602cd8b-bc49-4f09-9bf2-1ebd8375e03a
Commits

Reviewing files that changed from the base of the PR and between 26b9cb2 and 5733f62.

Files selected for processing: 5
  • quest/m1/interop-js-tone.md
  • quest/m1/test-flakes-2/README.md
  • quest/m1/test-flakes-2/interop-timestamp-gap.md
  • test/interop/clients/js-native/subscribe.ts
  • test/interop/interop.sh
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-24T17:47:28.385384Z 5733f62 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@kixelated

Copy link
Copy Markdown
Collaborator Author

Merge summary: required checks pass on head 5733f62, Codex reviewed this head with no findings (thumbs-up), no unresolved threads, and quest check --base origin/main passes (the finished quest interop-timestamp-gap and its links are removed). No changes were made during merge; no open decisions. Landing as a merge commit.

(Written by Claude Opus 5.5)

@kixelated
kixelated added this pull request to the merge queue Oct 24, 2026
Merged via the queue into main with commit 3818519 Oct 24, 2026
11 checks passed
@kixelated
kixelated deleted the quest/m1/test-flakes-2/interop-timestamp-gap branch October 24, 2026 19:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant