Skip to content
2 changes: 1 addition & 1 deletion doc/concept/hang.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ A few things the catalog can express beyond decoder config:
- **Reorder depth.** A video rendition may declare `reorder`, the most frames that can precede one in decode order and follow it in presentation, as H.264 `max_num_reorder_frames` counts them; `0` declares none, which is not the same as absent. The Rust H.264 and H.265 importers copy it from the SPS when the SPS declares one (progressive H.264 only), and `@moq/publish` declares `0`. The MPEG-TS, Matroska, and FLV exporters then derive each group's decode times from that group's frames alone, so exports that joined at different groups agree, and they end the export on a frame reordered deeper than declared or a depth the codec level cannot reach. Without it they infer decode times from the timestamps they have read, as before. A publisher never changes it once declared.
- **Colour.** A video rendition may carry `color`: the H.273 primaries, transfer, matrix, and range, plus the HDR10 mastering display and content light levels, so a renderer can set up HDR without parsing the bitstream. The Rust importers fill it from the SPS VUI, AV1 sequence header, and VP9 header, then fill gaps from a container's `colr` box or Matroska `Colour` element; the stream wins a disagreement. The HDR10 values come from H.264 and H.265 SEI or AV1 metadata OBUs in MPEG-TS and raw elementary streams, an AV1 track's av1C, or Matroska, but not yet from H.264 or H.265 in fMP4 or FLV. Nothing in `@moq/watch` reads it yet.
- **Spatial media.** A video rendition may carry `projection` (flat `rectilinear`, or `equirectangular` with the degrees of longitude it spans, 180 for VR180) and `stereo` (`mono`, `left-right`, or `top-bottom` frame packing), and an audio rendition may carry `layout: "ambisonic"` for AmbiX channels. Readers treat a value they don't recognize as absent. Nothing in `@moq/watch` renders them yet.
- **Disabled renditions.** An audio or video rendition with `enabled: false` has no frames coming, and a player must not select it. Pausing a rendition in `@moq/publish` keeps it in the catalog this way rather than removing it, so a mute is a one-field update; a disabled video rendition encodes one black keyframe first, so a player that predates the field shows black instead of a frozen picture. The field is written only when false. The older video `stalled` field is no longer written, and readers ignore it.
- **Disabled renditions.** An audio or video rendition with `enabled: false` has no frames coming, and a player should not select it. A player may keep a disabled rendition selected, and subscribed, while no enabled alternative exists, so a re-enable plays without a resubscribe; `@moq/watch` does. Pausing a rendition in `@moq/publish` keeps it in the catalog this way rather than removing it, so a mute is a one-field update; a disabled video rendition encodes one black keyframe first, so a player that predates the field shows black instead of a frozen picture. The field is written only when false. The older video `stalled` field is no longer written, and readers ignore it.
- **Preference.** A video rendition may carry a signed `preference`, default `0`. A player keeps only the highest preference it can decode, then picks by size and bitrate within it, as do the single-rendition egresses (RTMP play, WHEP, FLV, the transcoder's source). A publisher that adds its own H.264 copy of an H.265 source marks the copy `-1`, so only a viewer that can't decode H.265 pulls it. Picking a rendition by name still overrides it. The field is written only when nonzero.
- **HLS records.** A video or audio rendition may carry an `hls` field naming a small track of HLS segment records, each segment's first group and pts, plus the nominal segment duration. Each rendition numbers its own segments, and an edge serves a rendition's segment `n` from its record `n`; segments line up across renditions wherever their keyframes do. The records are what let the [HLS gateway](/bin/hls) build playlists without subscribing to media; a rendition without them gets no playlist. Rust publishers (`moq-mux` and everything built on it) publish them by default; `@moq/publish` does not yet. A reader that cannot decode the field ignores it.
- **Clock.** The optional root `clock` maps PTS zero to wall time so every media track and every HLS record share one fixed epoch after timescale conversion. Publish independent sources as sibling broadcasts under one prefix, such as `robot/front.hang` and `robot/rear.hang`; each broadcast's audio and video entries are renditions of that source. In one Rust publisher, pass the same `moq_mux::Clock` to each `catalog::Config::with_clock`; JS publishers on one page already share the page clock. Consumers discover the siblings by prefix and align their timestamps through the advertised wall mapping; `@moq/watch` plays one broadcast at a time.
Expand Down
5 changes: 3 additions & 2 deletions drafts/draft-lcurley-moq-hang.md
Original file line number Diff line number Diff line change
Expand Up @@ -774,7 +774,8 @@ For example, with video at 20 millisecond intervals and audio at 15 written as v

### enabled {#field-enabled}
The `enabled` field says whether a consumer may select an audio or video rendition; a text rendition does not carry it.
When `enabled` is false, no frames are coming, and a consumer MUST NOT select the rendition.
When `enabled` is false, no frames are coming, and a consumer SHOULD NOT select the rendition.
A consumer MAY keep a rendition it already selected while that rendition is disabled and no enabled one it can decode remains, so a re-enable needs no new subscription.
If absent, `enabled` defaults to true; a publisher SHOULD only write it when false.

Earlier versions defined a video `stalled` field instead.
Expand Down Expand Up @@ -1149,7 +1150,7 @@ This document has no IANA actions.
- Added the optional `delay` rendition field: how far a rendition's timestamps trail those other renditions already wrote, never lowered once advertised and never subtracted across renditions.
- Recommended namespaced keys for application root sections.
- Added the optional `delay` field to `json` and `binary` track entries, measured only from payloads stamped with their capture time on the broadcast clock.
- Replaced the video `stalled` field with an optional `enabled` field on audio and video renditions. A consumer MUST NOT select a disabled rendition and ignores `stalled`. A consumer that predates `enabled` keeps selecting a disabled rendition.
- Replaced the video `stalled` field with an optional `enabled` field on audio and video renditions. A consumer SHOULD NOT select a disabled rendition, MAY keep one it already selected while no enabled alternative remains, and ignores `stalled`. A consumer that predates `enabled` keeps selecting a disabled rendition.
- Audio `sampleRate` and `numberOfChannels` describe the decoded output, so HE-AAC names its SBR rate and parametric stereo names two channels.
- Added an optional `track` field to video, audio, text, `json`, and `binary` entries naming the track in its broadcast; the entry's key is its ID and names the track when `track` is absent. A consumer MUST reject a catalog using one ID for both a video and an audio rendition.
- Defined video `displayAspectWidth` and `displayAspectHeight` as the whole picture's display aspect, not the sample aspect.
Expand Down
3 changes: 2 additions & 1 deletion js/hang/src/catalog/audio.ts
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,8 @@ const AudioConfigFields = z.object({
bitrate: z.optional(u53Schema),

// Whether this rendition may be selected. When false, no frames are coming and a consumer
// must not select it. Default: true, so publishers only write it when false.
// should not select it, though one may keep it selected while no enabled alternative exists.
// Default: true, so publishers only write it when false.
enabled: z.optional(z.boolean()),

// The maximum delay between a frame being ready and the publisher flushing it, in whole
Expand Down
3 changes: 2 additions & 1 deletion js/hang/src/catalog/video.ts
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,8 @@ export const VideoConfigSchema = z.object({
bitrate: z.optional(u53Schema),

// Whether this rendition may be selected. When false, no frames are coming and a consumer
// must not select it. Default: true, so publishers only write it when false.
// should not select it, though one may keep it selected while no enabled alternative exists.
// Default: true, so publishers only write it when false.
enabled: z.optional(z.boolean()),

// Higher is preferred: a player selects among the highest preference it can decode, then by
Expand Down
59 changes: 56 additions & 3 deletions js/watch/src/audio/decoder.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -678,24 +678,77 @@ describe("Decoder across a rendition's absence", () => {
}
});

it("deselects a disabled rendition and keeps one graph across disable and enable", async () => {
it("holds a disabled rendition and keeps one subscription across mute and unmute", async () => {
const playback = await play(Milli(100));
try {
await playback.play(12);
expect(playback.track()).toBe("audio");

playback.disable();
await microtasks();
expect(playback.track()).toBeUndefined();
expect(playback.track()).toBe("audio");
expect(contexts).toHaveLength(1);

playback.restore();
await microtasks();
expect(playback.track()).toBe("audio");
await playback.play(6);
const before = playback.inserted().length;
const timestamps = await playback.play(6);

expect(contexts).toHaveLength(1);
expect(contexts[0].calls.close).toBe(0);
expect(playback.subscriptions()).toBe(1);
expect(playback.inserted().slice(before)).toEqual(timestamps);
} finally {
playback.close();
}
});

it("suspends a held muted tile once drained, and resumes on unmute", async () => {
const playback = await play(Milli(100));
try {
await playback.play(12);
const [ring] = playback.rings();
expect(ring.read([new Float32Array(128), new Float32Array(128)])).toBe(128);
await microtasks();

playback.disable();
await microtasks();
const [context] = contexts;
expect(context.calls.suspend).toBe(0);

expect(drain(ring)).toBeGreaterThan(0);
await microtasks();
expect(context.calls.suspend).toBe(1);
expect(context.state).toBe("suspended");
expect(playback.track()).toBe("audio");

playback.restore();
await microtasks();
expect(context.state).toBe("running");
expect(playback.subscriptions()).toBe(1);
} finally {
playback.close();
}
});

it("drops a held rendition once it is removed", async () => {
const playback = await play(Milli(100));
try {
await playback.play(12);
playback.disable();
await microtasks();
expect(playback.track()).toBe("audio");

playback.remove();
await microtasks();
expect(playback.track()).toBeUndefined();

// Its return is a fresh selection, with a fresh subscription.
playback.restore();
await microtasks();
expect(playback.track()).toBe("audio");
await playback.play(6);
expect(playback.subscriptions()).toBe(2);
} finally {
playback.close();
Expand Down
12 changes: 8 additions & 4 deletions js/watch/src/audio/decoder.ts
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,9 @@ export class Decoder {
const startup = this.#signals.computed((effect) => {
if (!effect.get(this.in.enabled)) return undefined;
if (effect.get(this.source.pending)) return null;
if (!effect.get(this.source.out.config)) return undefined;
// A held disabled rendition has no frames coming, so startup does not wait on it.
const config = effect.get(this.source.out.config);
if (!config || config.enabled === false) return undefined;
const span = effect.get(this.#decodeBuffered)[0];
if (!span) return null;
// Deep targets stay encoded, so count the audio queued past the decoded span. Sum it across
Expand All @@ -200,9 +202,11 @@ export class Decoder {
const config = effect.get(this.source.out.config);
return config ? playbackIdentity(config) : undefined;
});
this.#idle = this.#signals.computed(
(effect) => effect.get(this.source.out.config) === undefined && effect.get(this.#out.stalled),
);
// A held disabled rendition idles too, so a muted tile suspends its AudioContext.
this.#idle = this.#signals.computed((effect) => {
const config = effect.get(this.source.out.config);
return (!config || config.enabled === false) && effect.get(this.#out.stalled);
});

this.#signals.run(this.#runSubscribeMaxDelay.bind(this));
this.#signals.run(this.#runShape.bind(this));
Expand Down
130 changes: 129 additions & 1 deletion js/watch/src/audio/source.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,13 @@ import { Signal } from "@moq/signals";
import type { Broadcast } from "../broadcast";
import { Source } from "./source";

function config(codec: string): Catalog.AudioConfig {
function config(codec: string, fields: Record<string, unknown> = {}): Catalog.AudioConfig {
return Catalog.AudioConfigSchema.parse({
codec,
sampleRate: 48000,
numberOfChannels: 2,
container: { kind: "legacy" },
...fields,
});
}

Expand Down Expand Up @@ -66,3 +67,130 @@ test("a rejected replacement catalog clears previously available audio", async (
source.close();
}
});

test("skips a disabled rendition on the initial selection", async () => {
const state = broadcast({ mic: config("opus", { enabled: false }) });
const source = new Source({ broadcast: state.broadcast, supported: async () => true });
try {
await settled(source);
expect(source.out.track.peek()).toBeUndefined();

state.catalog.set({ audio: { renditions: { mic: config("opus") } } });
await settled(source);
expect(source.out.track.peek()).toBe("mic");
} finally {
source.close();
}
});

test("holds the selected rendition while every rendition is disabled", async () => {
const state = broadcast({ mic: config("opus") });
const source = new Source({ broadcast: state.broadcast, supported: async () => true });
try {
await settled(source);
expect(source.out.track.peek()).toBe("mic");

state.catalog.set({ audio: { renditions: { mic: config("opus", { enabled: false }) } } });
await settled(source);
expect(source.out.track.peek()).toBe("mic");
expect(source.out.config.peek()?.enabled).toBe(false);

state.catalog.set({ audio: { renditions: { mic: config("opus") } } });
await settled(source);
expect(source.out.track.peek()).toBe("mic");
expect(source.out.config.peek()?.enabled).toBeUndefined();
} finally {
source.close();
}
});

test("an enabled alternative takes over from a disabled rendition", async () => {
const state = broadcast({ mic: config("opus") });
const source = new Source({ broadcast: state.broadcast, supported: async () => true });
try {
await settled(source);
expect(source.out.track.peek()).toBe("mic");

state.catalog.set({
audio: { renditions: { mic: config("opus", { enabled: false }), backup: config("opus") } },
});
await settled(source);
expect(source.out.track.peek()).toBe("backup");
} finally {
source.close();
}
});

test("drops a held rendition once it is removed", async () => {
const state = broadcast({ mic: config("opus") });
const source = new Source({ broadcast: state.broadcast, supported: async () => true });
try {
await settled(source);
state.catalog.set({ audio: { renditions: { mic: config("opus", { enabled: false }) } } });
await settled(source);
expect(source.out.track.peek()).toBe("mic");

state.catalog.set({ audio: { renditions: { other: config("opus", { enabled: false }) } } });
await settled(source);
expect(source.out.track.peek()).toBeUndefined();
expect(source.out.config.peek()).toBeUndefined();
} finally {
source.close();
}
});

test("does not carry a hold to another broadcast", async () => {
const first = broadcast({ mic: config("opus") });
const current = new Signal<Broadcast | undefined>(first.broadcast);
const source = new Source({ broadcast: current, supported: async () => true });
try {
await settled(source);
first.catalog.set({ audio: { renditions: { mic: config("opus", { enabled: false }) } } });
await settled(source);
expect(source.out.track.peek()).toBe("mic");

// The same catalog on another broadcast probes nothing new, so wait on a timer, not the probe.
current.set(broadcast({ mic: config("opus", { enabled: false }) }).broadcast);
await new Promise((resolve) => setTimeout(resolve, 0));
expect(source.out.track.peek()).toBeUndefined();
} finally {
source.close();
}
});

test("does not hold an enabled selection into another broadcast's disabled rendition", async () => {
const current = new Signal<Broadcast | undefined>(broadcast({ mic: config("opus") }).broadcast);
const source = new Source({ broadcast: current, supported: async () => true });
try {
await settled(source);
expect(source.out.track.peek()).toBe("mic");

current.set(broadcast({ mic: config("opus", { enabled: false }) }).broadcast);
for (let i = 0; i < 5; i++) await new Promise((resolve) => setTimeout(resolve, 0));
expect(source.out.track.peek()).toBeUndefined();
expect(source.out.config.peek()).toBeUndefined();
} finally {
source.close();
}
});

test("holds on a broadcast whose renditions match the previous one", async () => {
const current = new Signal<Broadcast | undefined>(broadcast({ mic: config("opus") }).broadcast);
const source = new Source({ broadcast: current, supported: async () => true });
try {
await settled(source);

// The new probe publishes an equal `available`, so only its broadcast tag changes.
const next = broadcast({ mic: config("opus") });
current.set(next.broadcast);
await settled(source);
expect(source.out.track.peek()).toBe("mic");

next.catalog.set({ audio: { renditions: { mic: config("opus", { enabled: false }) } } });
await settled(source);
expect(source.out.track.peek()).toBe("mic");
expect(source.out.config.peek()?.enabled).toBe(false);
} finally {
source.close();
}
});
Loading
Loading