Skip to content

test(interop): the browser publisher sends its tone from a file, not a fake mic - #6647

Merged
kixelated merged 4 commits into
mainfrom
quest/m1/interop-js-tone
Oct 25, 2026
Merged

kixelated merged 4 commits into
mainfrom
quest/m1/interop-js-tone

Conversation

@kixelated

@kixelated kixelated commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Quest: quest/m1/interop-js-tone.md (deleted here).

Summary

The scheduled Interop run on 2026-10-23 (run 39958710886) failed js -> cells on drafts 18, 20, and 22 with "wrong tone: channel 0 carries ~440 Hz, not 440 Hz" at 0.10 to 0.77 of its energy.

Cause. The browser publisher captured the tone from Chromium's fake microphone (--use-file-for-fake-audio-capture) through a MediaStreamAudioSourceNode and the capture AudioWorklet. Chromium's fake capture and output devices run on timers that skip a 10 ms tick when they wake up late (FakeAudioWorker::DoRead jumps to the next interval instead of catching up). On a runner short of CPU, the captured tone then shifts by a fraction of a period, while the worklet's timestamps, which count rendered samples, stay contiguous. Every subscriber decodes the splice exactly: tone.py sees no gap, only a 440 Hz tone whose phase steps several times inside its 400 ms window, so its Goertzel share drops. The browser subscriber's analyser, which judges a moving window, passed. This is not a decoder or a relay bug, and not the max-age mechanism in moq-dev/moq#6658: the jumps showed up inside the publisher's own capture worklet before encoding, with no timestamp gap anywhere.

Why it started on 2026-10-23. Nothing in js/publish changed. moq-dev/moq#6284 (every subscriber checks the tone) landed between the passing 2026-10-21 run and the failing one. The splices predate it. The --versions ietf job runs last, after about 45 minutes of the full matrix, which is when the runner is most loaded. moq-dev/moq#6299 is not involved. Locally I bisected between the two scheduled runs' commits: at dbf06bdcb3, from before the native tone check, 5 of 5 runs passed. At 4381af6f07, the failing scheduled run's head, 5 of 5 runs failed the tone, 1 to 9 cells per run.

Fix. The browser publisher no longer uses fake devices. interop.sh generates a 10 s H.264 and 16-bit PCM .mov holding the picture and the tone. The driver serves it, and the page hands it to <moq-publish>'s file source, which decodes it with mediabunny and paces it in real time. The tone check is unchanged. This also exercises the file source end to end in every subscriber. Without media, the publish role keeps its fake-device path for media.ts's capture-permission case.

Reproduction

just test interop --publishers js --subscribers rust,python,go,cpp,js,js-native-node,js-native-bun,c,gst --versions moq-transport-18,moq-transport-22, pinned to 2 CPUs with taskset, 5 runs each (18 js -> cells per run):

Commit Runs with a wrong tone Cells failed
4381af6f07 (2026-10-23 scheduled head) 4 of 5 1, 8, 9, 6
main (57b0b32bbd) 1 of 2 (stopped early) 8
this branch 0 of 5 0

Unpinned, every commit passes. A frame-by-frame phase readout of a failing decode showed a steady offset, then a step of about 300 to 900 us partway through. An instrumented publisher logged the same phase jumps from inside its capture worklet at the matching times. A standalone page reading the fake microphone through both an AudioWorklet and MediaStreamTrackProcessor showed jumps on both paths, even with the CPU idle.

Decisions made without asking

  • Fixed the test publisher rather than js/publish's capture. The capture correctly stamps what Web Audio hands it, and a real microphone has a real clock. The fault is Chromium's fake device timers, which we can't change. Publishing a file is a supported <moq-publish> source.
  • serve() in the Playwright harness takes an optional map from path to file, so the driver can serve the generated media beside the built page.

Public API and wire impact

None. Test harness only.

Checks

  • just check: passed.
  • just test interop --all: passed (164 PASS lines).
  • just test interop --all --versions ietf: passed on drafts 14 to 22 (312 PASS lines), after merging main.

Follow-ups

  • The tone check in the browser subscriber judges a moving window and missed these splices. That is fine for playback, so no quest is suggested.

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

kixelated and others added 3 commits October 24, 2026 09:17
…a fake mic

Chromium's fake capture and output devices run on timers that skip a late
10 ms tick. Under CPU starvation Web Audio then splices the captured tone
while the capture worklet's sample-count timestamps stay contiguous, so every
native subscriber's tone check (added in #6284) reads ~440 Hz at a fraction
of its energy. The browser publisher now decodes and loops a generated
H.264 + PCM file through <moq-publish>'s file source instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title The JS interop publisher's tone survives every draft test(interop): the browser publisher sends its tone from a file, not a fake mic Oct 24, 2026
@kixelated
kixelated marked this pull request as ready for review October 24, 2026 18:22
@coderabbitai

coderabbitai Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 20 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 7b0daa95-7412-4109-b005-26ddafa76b38
Commits

Reviewing files that changed from the base of the PR and between 1092357 and 24fa65a.

Files selected for processing: 7
  • quest/m1/README.md
  • quest/m1/interop-js-tone.md
  • test/interop/README.md
  • test/interop/clients/js/driver.ts
  • test/interop/clients/js/harness.ts
  • test/interop/clients/js/src/setup.ts
  • test/interop/interop.sh
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-25T00:12:17.625072Z 24fa65a New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@kixelated

Copy link
Copy Markdown
Collaborator Author

Adversarial review of the "outside our code" claim, with a standalone headless Chromium page reading the fake microphone (440 Hz WAV) through both MediaStreamTrackProcessor and an AudioWorklet, measuring each 10 to 20 ms block's phase step against its sample count and its timestamp step.

Condition MediaStreamTrackProcessor AudioWorklet
Idle one step of -626 us at the WAV's 10 s loop seam the same seam step, nothing else
Pinned to 2 CPUs beside 16 busy loops, 20 s a 320 ms timestamp jump with the content continuous (-8 us) splices of -909 us (exactly one 10 ms buffer at 440 Hz) and -590 us, no timestamp gap; ctx.currentTime reached 19.75 s over 20.0 s of page time

So:

  • The fake device skipping a tick does not splice the content. It delivers fewer buffers, and MediaStreamTrackProcessor's capture timestamps show the gap. The splice comes from MediaStreamAudioSourceNode bridging the capture into the AudioContext's render clock, which drops or pads whole buffers when the two disagree. That is still Chromium, and our worklet can't see it, so the fix belongs in the test: with a processor-based capture, the same starvation would fail the cell as a timestamp gap instead of a wrong tone.
  • The "jumps on both paths even with the CPU idle" in the description look like Chromium's FileSource loop seam, not the tick skips.
  • It isn't fully harmless in production, though. js/publish stamps audio by counting render-clock samples while video carries capture timestamps, so audio drifts against video by however far the render clock strays from the capture clock: about 250 ms per 20 s on a starved fake output, and crystal drift (tens of ppm) on real hardware. That is a separate js/publish design question, not this PR's problem.

Coverage after this PR: the capture path is still exercised. media.ts (scheduled just test media, and nightly with a Firefox player) publishes its fixture's oscillator through a MediaStreamTrack into Publish.Audio.Capture and checks tone and A/V sync in a browser player. Its capture-denial case goes through getUserMedia on the fake devices. capture-channel-count.browser.ts checks the worklet's format overrides and timestamp gaps. What's lost is native subscribers decoding a captured stream, and Chromium's fake devices can't give that reliably.

Local checks on 649c4dd: just check passed, and just test interop --publishers js --subscribers rust,python,js,js-native-node --versions moq-transport-18,moq-transport-22 passed both unpinned and pinned to 2 CPUs (all 8 js -> cells each time).

(Written by Claude Opus 5.5)

@kixelated

Copy link
Copy Markdown
Collaborator Author

Merging 649c4dd with the maintainer's approval. The non-Grok review of this head is clean, every finding is fixed or answered, and the pre-land quest check passes against origin/main.

(Written by Claude Opus 5.5)

@kixelated
kixelated added this pull request to the merge queue Oct 24, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Oct 24, 2026
@kixelated

Copy link
Copy Markdown
Collaborator Author

Merged origin/main again after the merge queue dropped this for a conflict. The only conflict was quest/m1/interop-js-tone.md: main dropped its Related link to the finished interop-timestamp-gap quest (moq-dev/moq#6658), and this PR deletes the quest, so it stays deleted. test/interop/interop.sh and quest/m1/README.md merged cleanly; the rewritten quest/m1/fmp4-edit-lists.md from moq-dev/moq#6686 already sequences itself after this PR and needs no change. The PR's own diff is unchanged. just check and the pre-land quest check pass on 24fa65a.

(Written by Claude Opus 5.5)

@kixelated
kixelated added this pull request to the merge queue Oct 25, 2026
Merged via the queue into main with commit 68ed281 Oct 25, 2026
10 checks passed
@kixelated
kixelated deleted the quest/m1/interop-js-tone branch October 25, 2026 00:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant