Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions doc/bin/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,7 @@ MPEG-TS export is covered in the [MPEG-TS guide](/bin/ts#export).
```bash
moq --connect https://relay.example.com/anon --broadcast my-stream.hang play
moq ... play --delay 500ms # fix the delay instead of measuring it
moq ... play --stretch 200ms # let audio trail the clock by at most 200 ms
moq ... play --video off # audio only, without a window
moq ... play --video off --metrics play.ndjson # log what the speaker played
```
Expand All @@ -188,10 +189,14 @@ names the decoder each track opened.
Playback starts at the newest group and trails the live edge by `--delay`. The
default, `auto`, sizes the buffer to how unevenly audio arrives, with the same
[algorithm](/concept/audio-jitter) as the browser player. A duration fixes the
delay and is also how long a stalled group is waited on before it is skipped.
Audio and video play from one clock and stay in sync while both arrive on time;
a track that stalls plays late and catches up by playing faster, rather than
holding the other one back.
delay instead. Audio and video play from one clock and stay in sync while both
arrive on time. After a stall, video shows each late frame as it arrives, while
audio slows down and falls behind the clock rather than dropping what arrives
late, then speeds up until it is back on it. `--stretch` (default 2 s) bounds
how far audio may trail before it skips ahead, and a stalled group is waited on
for the delay plus the stretch before it is skipped. `--stretch 0` plays
received audio unaltered: late audio is skipped, an underrun is silence, and
drift against the publisher's clock is corrected by a small skip or pad.
Each role follows the catalog, switching rendition when the publisher retires
the one playing. With `--video off`, `--metrics <PATH>` writes a JSON line every
250 ms with the playhead, the delay, every gap and stall the speaker hit, and
Expand Down
41 changes: 27 additions & 14 deletions doc/concept/audio-jitter.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,9 +46,9 @@ group is observed when the stall clears. A missing group, one that has delivered
no frame, is waited for only the playout delay plus any lookahead, since past that
it would play late: a reordered backlog's holes waited out to the whole budget
would delay the frames behind them, which the estimator would measure as
jitter. A caller that estimates keeps the budget
at least as long as the estimate's ceiling, which is why `moq play` waits 2 s on
a stalled group under `--delay auto`. The observation is also only as honest as
jitter. A caller that estimates keeps the budget past the target, which is why
`moq play` waits on a stalled group for the delay plus its `--stretch`, 2 s by
default. The observation is also only as honest as
the caller is prompt: `read()` is timed when it resolves, so a player drains it
as packets come and lets the playback buffer hold the target, rather than
pacing reads to the speaker.
Expand All @@ -58,10 +58,16 @@ drop is not the receiver's own doing. The subscription's max delay drops groups
before any container sees them, at the relay and in the receiver's own transport
subscriber, so a subscription cut to the target hides every frame later than
the target: the estimate creeps up one bucket at a time while the frames it
should have measured go unplayed. In automatic mode both implementations
subscribe with at least the ceiling, 2 s: `moq play --delay auto` through its
age budget, and the browser through the audio subscription alone, since its
container consumer observes each frame before applying the local budget.
should have measured go unplayed. Both implementations subscribe with the delay
plus the player's `stretch` setting: `moq play` through its age budget, and the
browser through the audio subscription alone, since its container consumer
observes each frame before applying the local budget. The 2 s default reaches
the estimate's own ceiling, so any lateness the estimator can measure is
observed at once. A smaller stretch hides frames later than the target plus the
stretch, so a jump in lateness larger than the stretch is learned in steps of
about the stretch; a zero stretch cuts a live subscription to the target and
accepts the one-bucket creep. A buffered browser player keeps its subscription
at least the delay plus its lookahead, whatever the stretch.
Browser video keeps the shared budget, because a longer one fetches whole stale
groups, so a broadcast with no audio measures only the lateness that budget
lets through.
Expand Down Expand Up @@ -676,9 +682,9 @@ At each window boundary a player decides on five inputs:
- `lag`: how far the playhead trails the shared clock, negative when it plays
ahead of it.
- `bound`: how far audio may trail the clock, which is the subscription's max
delay less the playout delay. It is zero at a fixed live delay, the lookahead
for a buffered player, and at `auto` the shared max delay raised to the
estimator's 2 s ceiling, less the delay.
delay less the playout delay: the player's `stretch` setting (2 s by
default), or for a buffered player the larger of it and the lookahead. A zero
stretch turns the stretch off; see [below](#skips-underruns-and-silence).

`level` passes through NetEq's `BufferLevelFilter` once per window: three of its
10 ms steps at the factor the target selects, 251, 252, 253, or 254 in 256ths
Expand Down Expand Up @@ -717,10 +723,9 @@ and slowing down lets audio fall behind by at most `bound`, or brings a
playhead ahead of the clock back onto it. These are NetEq's thresholds and
rates, untuned. A moved target moves the clock, so a rise leaves audio ahead of
it and a fall leaves audio behind, and both become ordinary slowing down and
speeding up: nothing parks in silence on a rise, and in `auto` nothing skips
buffered audio on a fall, since the bound, the max delay less the target,
always covers a fall of the target. A fixed delay lowered at runtime by more
than its bound and one window skips to the bound, as any lag past it does.
speeding up: nothing parks in silence on a rise, and nothing skips buffered
audio on a fall shorter than the bound and one window. A fall past that skips
to the bound, as any lag past it does.

Without a clock reading the target is where the playhead belongs, so
`lag = level - target`. The browser then never skips; native playback bounds
Expand Down Expand Up @@ -763,6 +768,14 @@ next frame while what it was given, landed or still decoding, reaches less than
its floor and one window past the playhead, so the ring always holds a full
window to stretch, and a hole in the media plays as silence.

A zero `stretch` turns the stretch off, for content where a pitch-period splice
is audible, such as music. Every window holds, so received audio plays
unaltered, and the bound is zero: a playhead trailing the clock by more than one
window skips onto it, and one more than a window ahead of it pads silence until
the clock reaches it. Late audio is skipped, an underrun is silence, and drift
between the publisher's clock and the device's is corrected by a skip or a pad
of about one window.

### Pitch search and admission

Use channel zero as NetEq does. All channels share the selected seam and lag,
Expand Down
31 changes: 24 additions & 7 deletions doc/lib/js/watch.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ in sync at the latency you ask for.
| `paused`, `muted`, `volume` | The usual player controls. |
| `delay` | How far playback trails the live edge: `"auto"` (the default, sized from how late frames arrive; see [audio jitter](/concept/audio-jitter)), a duration like `"300ms"`, or `"instant"` to paint video as it decodes, with no audio. |
| `lookahead` | How far ahead of the live edge media may run; see [buffered playback](#buffered-playback). |
| `stretch` | How far audio may trail the clock before it skips ahead: a duration like `"2s"` (the default), or `"0"` to turn the time stretch off; see [catching up](#catching-up). |
| `captions` | The caption track to show. |
| `visible` | Only download video while the element is on screen (or near it). |
| `backend` | How the canvas is drawn; see [@moq/video](/lib/js/video). `auto` (default), `webgpu`, or `2d`. |
Expand Down Expand Up @@ -218,16 +219,31 @@ leave the connection's estimate unclaimed. Without a claim, video selects the be
`video.target` allows. To stop the relay probing while still adapting, keep the claim and call
`connection.viewers.setTarget(0)`.

## Catching up

After a stall, video shows each late frame as it arrives and is live again at
once. Audio cannot skip without losing what arrives late, so it slows down
slightly when the audio received ahead of it runs low, falls behind the clock,
and speeds back up once it recovers until it is back on it. `stretch` bounds
how far it may trail: only audio later than that is skipped, and the audio
subscription waits on a stalled group for `delay + stretch`, so the late audio
still arrives. A voice app might ask for `stretch="200ms"`, a sports stream for
the default 2 s, whether `delay` is `auto` or fixed.

`stretch="0"` turns the time stretch off, for content where a pitch-period
splice is audible, such as music: received audio plays unaltered, late audio
is skipped, an underrun is silence, and drift between the publisher's clock
and the device's is corrected by a small skip or pad.

## Buffered playback

By default the player minimizes latency: audio plays at the delay, slows down
slightly when the audio received ahead of it runs low, speeds back up once it
recovers, and skips ahead only audio trailing by more than the subscription
still delivers. Content produced faster than real time, such as a TTS
By default the player minimizes latency: playback trails the live edge by the
delay, and audio catches up after a stall as [above](#catching-up). Content produced faster than real time, such as a TTS
response emitted in one burst with future timestamps, wants the opposite. Set
`lookahead` to how far ahead it may run and it plays through at the encoded pace.
If a stall leaves audio behind, it catches up by playing faster, as live
playback does, and skips only what trails by more than the lookahead:
playback does, and skips only what trails by more than the larger of the
lookahead and `stretch`; a zero `stretch` still turns the stretch off:

```html
<moq-watch url="..." name="bot/tts.hang" delay="100ms" lookahead="30s"></moq-watch>
Expand All @@ -237,8 +253,9 @@ playback does, and skips only what trails by more than the lookahead:
`delay`, since playback only skips ahead once media sits more than
`delay + lookahead` past the playhead.

Durations need a unit; a bare number is rejected. From script, the `delay` and
`lookahead` properties take a `Duration` from `@moq/clock`, such as
Durations need a unit; a bare number other than zero is rejected. From script,
the `delay`, `lookahead`, and `stretch` properties take a `Duration` from
`@moq/clock`, such as
`el.delay = Duration.millis(300)`. Audio holds the lookahead
as encoded frames, so a large one is cheap; video waits as decoded pictures,
which hold decoder memory, so a long video lookahead is not. The first video
Expand Down
1 change: 1 addition & 0 deletions js/watch/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ The simplest way to watch a stream:
| `follow` | boolean | true | Follow the broadcast's announcements: wait for one before subscribing, and move to each restart. Ignored when the relay does not support broadcast discovery. |
| `delay` | `auto`, duration, `instant` | `auto` | Distance from the live edge. `instant` paints frames as they decode and disables audio. |
| `lookahead` | duration | `0ms` | How far future-dated media may run ahead of the live edge. Not a jitter buffer: live streams tune `delay`. |
| `stretch` | duration | `2000ms` | How far audio may trail the clock after a stall before it skips ahead. `0` turns the time stretch off, so received audio plays unaltered. |
| `captions` | string | off | Text rendition to render. |
| `catalog-format` | hang, msf, manual | auto-detected | The catalog format; detected from the name suffix unless set. |
| `echo` | path segment | - | The viewer's feedback name: once the catalog solicits feedback, publish `<echo path>/<name>.echo` reporting what each rendition received and played. |
Expand Down
7 changes: 5 additions & 2 deletions js/watch/api/element.api.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ import { Video } from "@moq/watch";

export default class extends HTMLElement {
constructor();
attributeChangedCallback(name: "delay" | "lookahead" | "url" | "echo" | "name" | "follow" | "volume" | "muted" | "paused" | "visible" | "backend" | "versions" | "announced" | "buffer" | "catalog-format" | "captions", oldValue: string | null, newValue: string | null): void;
attributeChangedCallback(name: "delay" | "lookahead" | "stretch" | "url" | "echo" | "name" | "follow" | "volume" | "muted" | "paused" | "visible" | "backend" | "versions" | "announced" | "buffer" | "catalog-format" | "captions", oldValue: string | null, newValue: string | null): void;
readonly audio: Audio.Decoder;
get backend(): Video.Backend;
set backend(value: Video.Backend);
Expand All @@ -37,6 +37,7 @@ export default class extends HTMLElement {
delay: Signals.Signal<Delay>;
lookahead: Signals.Signal<Clock.Duration>;
paused: Signals.Signal<boolean>;
stretch: Signals.Signal<Clock.Duration>;
text: {
target: Signals.Signal<string | undefined>;
};
Expand All @@ -63,12 +64,14 @@ export default class extends HTMLElement {
set muted(value: boolean);
get name(): Net.Path.Valid;
set name(value: string | Net.Path.Valid);
static observedAttributes: readonly ["url", "versions", "name", "paused", "volume", "muted", "visible", "backend", "follow", "announced", "buffer", "delay", "lookahead", "catalog-format", "captions", "echo"];
static observedAttributes: readonly ["url", "versions", "name", "paused", "volume", "muted", "visible", "backend", "follow", "announced", "buffer", "delay", "lookahead", "stretch", "catalog-format", "captions", "echo"];
get paused(): boolean;
set paused(value: boolean);
readonly renderer: Video.Renderer;
reset(): void;
readonly signals: Signals.Effect;
get stretch(): Clock.Duration;
set stretch(value: Clock.Duration);
readonly sync: Sync;
readonly text: Text.Source;
readonly textRenderer: Text.Renderer;
Expand Down
2 changes: 2 additions & 0 deletions js/watch/api/index.api.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,7 @@ export type PlayerInput = {
name: Signals.Getter<Net.Path.Valid>;
origin: Signals.Getter<Net.Origin.Reader | undefined>;
paused: Signals.Getter<boolean>;
stretch: Signals.Getter<Clock.Duration>;
text: {
container: Signals.Getter<HTMLElement | undefined>;
target: Signals.Getter<string | undefined>;
Expand Down Expand Up @@ -188,6 +189,7 @@ export class Sync {
export type SyncInput = {
delay: Signals.Getter<Delay>;
lookahead: Signals.Getter<Clock.Duration>;
stretch: Signals.Getter<Clock.Duration>;
};

export type SyncOutput = {
Expand Down
34 changes: 29 additions & 5 deletions js/watch/src/audio/decoder.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import * as Moq from "@moq/net";
import { Effect, Signal } from "@moq/signals";
import type { Broadcast } from "../broadcast";
import { supportCacheKey } from "../support-probe";
import { type Delay, Sync } from "../sync";
import { type Delay, STRETCH, Sync } from "../sync";
import { Decoder as VideoDecoder } from "../video/decoder";
import { Source as VideoSource } from "../video/source";
import { decoderConfigKey } from "./config";
Expand Down Expand Up @@ -341,7 +341,8 @@ async function play(initial: Delay, buffered = false, cmaf = false) {
Object.assign(supportProbes, { [supportCacheKey]: decoderConfigKey });
const source = new Source({ broadcast, supported: supportProbes });
const lookahead = new Signal(Duration.millis(buffered ? 1000 : 0));
const sync = new Sync({ delay, lookahead });
const stretch = new Signal(STRETCH);
const sync = new Sync({ delay, lookahead, stretch });
// The decoder's own switch, which a mute or pause turns off.
const decoding = new Signal(true);
const decoder = new Decoder({ source, sync, enabled: decoding });
Expand All @@ -363,6 +364,7 @@ async function play(initial: Delay, buffered = false, cmaf = false) {
supportProbes,
delay,
lookahead,
stretch,
sync,
push,
queued,
Expand Down Expand Up @@ -1711,6 +1713,28 @@ it.each([false, true])("releases shared startup across a hole in queued audio (C
}
});

// Audio may trail the clock by the stretch, or a buffered player's whole lookahead when that is longer,
// and a zero stretch turns the time stretch off in either mode.
for (const buffered of [false, true]) {
it(`bounds ${buffered ? "buffered" : "live"} audio's lag by the stretch`, async () => {
const playback = await play(Duration.millis(100), buffered);
try {
await playback.play(12);
const bound = () => worklets.at(-1)?.terms?.bound;
expect(bound()).toBe(2000);
playback.stretch.set(Duration.millis(300));
await microtasks();
expect(bound()).toBe(buffered ? 1000 : 300);
playback.stretch.set(Duration.ZERO);
await microtasks();
expect(bound()).toBe(0);
} finally {
playback.close();
await microtasks();
}
});
}

for (const cmaf of [false, true]) {
for (const buffered of [false, true]) {
it(`switches a running ${cmaf ? "CMAF" : "legacy"} player to ${buffered ? "buffered" : "live"} audio`, async () => {
Expand All @@ -1730,9 +1754,9 @@ for (const cmaf of [false, true]) {
(buffered ? [0, 20, 40, 60, 80] : [140, 160]).map((ms) => start + ms * 1000),
);

// Either ring keeps what it receives and plays onto the clock. Buffered audio may trail it
// by the lookahead; live audio at a fixed delay by nothing the subscription still delivers.
expect(worklets.at(-1)?.terms).toEqual({ latency: 100, target: 100, bound: buffered ? 1000 : 0 });
// Either ring keeps what it receives and plays onto the clock, which audio may trail by the
// larger of the lookahead and the default stretch, even at a fixed delay.
expect(worklets.at(-1)?.terms).toEqual({ latency: 100, target: 100, bound: 2000 });
} finally {
playback.close();
await microtasks();
Expand Down
Loading