diff --git a/README.md b/README.md index a3b91ff6..39f77474 100644 --- a/README.md +++ b/README.md @@ -300,7 +300,12 @@ broadcast through the first relay while that relay has a route to it, and through the failover otherwise, so a publisher restarted behind only the failover replaces the first relay's copy once that copy ends. It plays the best video and audio its catalog lists, switches when the catalog changes, and -comes back on its own when the publisher restarts. +comes back on its own when the publisher restarts. While it is offline, or +once a new stream replaces the one it played, it shows nothing until the new +stream's first picture, rather than freezing on the last one. A broadcast +without video but with an image, such as a studio's image or color published +independently, shows that image, and never reads as stalled, since it sends +a picture only when it changes. A broadcast source, a guest's included, plays on its publisher's timestamps, so its sound and picture stay in sync through network jitter and stalls. Its @@ -327,8 +332,9 @@ tagged `audio` and listed after the scenes. The Audio panel is a mixer. Each sound has a strip, with its gain, pan, mute, delay, and meter, which apply in every audio scene holding it, and a send to -each audio scene; each audio scene has a strip with its meter and a send to -each other audio scene. A source added from the Sources or Devices panel, or +each audio scene; each audio scene has a strip with its meter, which reads +it as **Listen** plays it, published sources included, and a send to each +other audio scene. A source added from the Sources or Devices panel, or pasted, is sent to the audio scene on program, so it is heard at once. **On program** picks the audio scene the broadcast carries, and switching it, or editing an audio scene, never drops or repeats audio. A recording holds @@ -543,7 +549,9 @@ before it expires, and is then renewed for as long as they stay, until you remove them. A guest's broadcasts become sources named after them (`alice`, `alice screen`) when they first appear, in no scene or audio scene: place them from the Items panel and send them to the audio scene on program in Audio -to put them on air. That includes a +to put them on air, or tick **Published independently** to have viewers fetch +them from the guest instead of through the program (see +[Published independently](#published-independently)). That includes a guest invited before a restart whose link still works. A guest's source is saved to the scene file but is not your edit: it stays out of History, so undo never removes it and a guest joining never loses what you could redo. A guest @@ -866,7 +874,7 @@ stream. Each camera, display, window, image, media file, color, microphone, and system audio also has a track of its own, so the studio can replay or record it on its -own. These tracks stay in the studio unless a source is [published independently](#published-independently): otherwise the program is all it publishes to a relay. They are listed in a local +own. These tracks stay in the studio unless a source is [published independently](#published-independently): otherwise the program and its composition are all it publishes to a relay. They are listed in a local catalog under the `live.kixel.studio` section's `sources`, by source name, each as a hang catalog of its own on the program's clock. @@ -921,25 +929,36 @@ where it can and software otherwise. ### Published independently -A source with a track of its own can be **Published independently**, a -checkbox in the Sources panel, off by default and kept in the scene file's -`published` list. Its track then goes to the relay beside the program, in the -same broadcast: the program's catalog lists it under the `live.kixel.studio` -section's `sources`, by source name, as a hang catalog of its own whose track -names start with `source/`. Viewers fetch it and place it over the program -themselves, as the [layer layout](#layer-layout) says. +A broadcast source, such as a guest's camera, or a source with a track of its +own can be **Published independently**, a checkbox in the Sources panel, off +by default and kept in the scene file's `published` list. Viewers then fetch +it as a broadcast of its own and place it over the program themselves, as the +[composition](#composition) says: + +- A local source's track goes to the relay as a broadcast of its own at + `<--broadcast>/source/`, with a hang catalog on the program's clock, + so its name is one segment of that path: no `/`, not `.` or `..`, and a `\` + only before another `\`. +- A broadcast source is already its publisher's, so viewers read it where the + studio does, which their relay must let them: a guest's, under the room. + The studio publishes nothing more of it. The program leaves a published source out, wherever it or a clone of it is -placed, and so does every audio scene's mix. The studio draws it only as -viewers will: over the program, in the Program panel and the program's -virtual camera, and over the scene in preview, so both show what viewers -see. It goes over every other item of its scene, and over the top layer, -however the scene stacks it. Recordings hold the program as published, -without it, and recording **each source too** gives a published camera, -display, window, or microphone a file of its own. Published and recorded -scenes, and a scene that shows the program, leave it out too. While viewers read its track, it claims its bitrate of the connection beside the program's, and its video encodes at what the connection grants it. Its track carries one rendition, the -plain capture before any filter or gain, and encodes for as long as it is -published, holding an encode session as one on program does. +placed, and so does every audio scene's mix the studio encodes: the +program's audio, recordings, and published scenes. People in the studio +still hear it as viewers do: **Listen** and guests' return feeds mix it in +where its audio scene reaches it, at its gain, mute, pan, and delay. The +studio draws it only as viewers will: over the program, in the Program +panel and the program's virtual camera, and over the scene in preview, so +both show what viewers see. It goes over every other item of its scene, and +over the top layer, however the scene stacks it. Recordings hold the program +as published, without it, and recording **each source too** gives a published +source a file of its own. Published and recorded scenes, and a scene that +shows the program, leave it out too. While viewers read a local source's +track, it claims its bitrate of the connection beside the program's, and its +video encodes at what the connection grants it. Its track carries one +rendition, the plain capture before any filter or gain, and encodes for as +long as it is published, holding an encode session as one on program does. Every player must reproduce how it is placed, so an item showing it takes no filter but opacity and no blend mode, nor do its source's filters or its @@ -947,74 +966,109 @@ keyframes, and no scene that another scene draws can place it. A source with its background removed cannot be published, since H.264 carries no transparency and its track would show the background. -#### Layer layout +#### Composition -While a source is published independently, the program's broadcast also -carries a JSON track, `live.kixel.studio.layers`, listed in its catalog's `json` -section. It is a snapshot: each change is a new group, stamped on the media -clock with the frame it takes effect on, so a viewer who joins late reads the -layout in force. The format is the studio's own until hang specifies a layer -layout: +Beside the program, the studio publishes a composition at +`<--broadcast>/composition`, a broadcast of its own whose catalog's `json` +section lists one JSON track, `live.kixel.studio.composition`, and whose +`media.clock` is the program's. The track is a snapshot: each change is a new +group, stamped on the program's media clock with the frame it takes effect on, +so a viewer who joins late reads the composition in force. A player that +knows no compositions plays the program at `<--broadcast>` as before. The +format is the studio's own until hang specifies a composition: ```json { - "version": 1, + "version": 2, "canvas": { "width": 1920, "height": 1080 }, "start": 10000000, + "program": { "broadcast": ".", "instance": "5c1f0e3a9b7d4e21a8f6c0d3b2e19f47" }, + "sources": { + "camera": { "broadcast": "source/camera" }, + "mic": { "broadcast": "source/mic", "sound": { "gain": 1.0, "delay": "80ms" } }, + "alice": { + "broadcast": "../guests/alice/camera.hang", + "instance": "9e4b7c21d0a3f6e85b1c2d4f7a9e0b36", + "offset": -1500000, + "sound": { "gain": 0.8, "pan": -0.5 } + } + }, "pictures": [ { - "item": { - "source": "camera", - "rect": { "x": 1440.0, "y": 760.0, "width": 400.0, "height": 225.0 }, - "visible": true, - "filters": [{ "kind": "opacity", "amount": 0.9 }], - "keyframes": [{ "at": "1s", "x": 600.0 }] - } - } - ], - "sounds": [{ "source": "mic", "audio": { "gain": 1.0, "muted": false, "delay": "80ms" } }] + "source": "camera", + "rect": { "x": 1440.0, "y": 760.0, "width": 400.0, "height": 225.0 }, + "filters": [{ "kind": "opacity", "amount": 0.9 }], + "keyframes": [{ "at": "1s", "x": 600.0 }] + }, + { "source": "alice", "rect": { "x": 40.0, "y": 760.0, "width": 400.0, "height": 225.0 } } + ] } ``` -- `version` is 1. A player refuses any other loudly and plays the program - alone. +- `version` is 2. A player refuses any other loudly and plays the program + alone. Version 1 was the layer layout, a track of the program's broadcast + naming its tracks. - `canvas` is the program's size, which pictures are placed on in pixels. - `start` is when the scene on program went on, in microseconds on the media clock: the moment its pictures' keyframes play from. +- `program` names the program's broadcast, and `sources` each source + published independently that a picture shows or the program's audio plays, + by source name. Each `broadcast` is relative to the composition's path, as + hang's references are, so `.` is the program and `..` walks up a segment. + An `instance` names the stream it describes, left out when the relay minted + the one the studio reads. +- A source's `offset`, in microseconds (0 if absent), is added to its + broadcast's timestamps to put them on the program's clock. A local source + is stamped on that clock, at 0. A guest keeps a timeline of its own, so its + offset is where the studio played it: a new composition follows it once it + moves more than 10 ms, and a guest who joins again, as a new instance, gets + a new one. A guest the studio shows nothing of, such as one offline, is + left out with its pictures until it shows again. A still, + such as another studio's published image, has no moment, so without sound + its offset is 0. +- A source's `sound`, when the audio scene on program holds it, is the + `gain`, `muted`, `pan`, and `delay` it plays at. +- Each source may carry a subscription `priority` (0 if absent) and whether a + viewer may `toggle` it off (`false` if absent). - `pictures` lists each item of the scene on program, then of the top layer, showing a published source, back to front over the program, shown or - hidden. Each `item` is a scene file item: its `source` names the source's - entry under the catalog's `live.kixel.studio` `sources` (the source a clone - shows), and its `crop` and `filters` fold in its source's own. -- `sounds` lists each published source the audio scene on program holds, - with the `gain`, `muted`, `pan`, and `delay` it plays at. -- Each picture and sound may carry a subscription `priority` (0 if absent) - and whether a viewer may `toggle` it off (`false` if absent). - -A take changes the layout on the frame the program starts showing the new -scene, so the published sources of the scene leaving go off and those of the -scene coming in go on at once, even through a transition. + hidden. Each is a scene file item: its `source` names a key of `sources` + (the source a clone shows), and its `crop` and `filters` fold in its + source's own. + +A take changes the composition on the frame the program starts showing the +new scene, so the published sources of the scene leaving go off and those of +the scene coming in go on at once, even through a transition. A published +scene named `composition` would publish where the composition is, so it does +not publish. #### Watching -`moq-studio --watch https://relay.example/anon/my-show` plays a studio -broadcast in a window as a viewer sees it: the program, with each picture of -its layer layout over it and each sound mixed with it. The URL's path names -the broadcast, as `moq play`'s does, and `--connect-token` and the `--quic-*` -flags apply to it; `--connect`, `--failover`, and `--room` are refused -beside it. It is a prototype for settling the layout's format, which goes -once `moq play` plays layers. - -- Each layout takes effect when the program plays the frame it is stamped - with, and its pictures' keyframes play from its `start`. -- Each source plays its own track on the program's clock: a picture shows - when the program's picture stamped alike does, and a sound plays with the - program's sound stamped alike, at its `gain`, `muted`, `pan`, and `delay`. - A frame too late for its moment is dropped. -- A broadcast without a layout, or with a layout of another `version`, plays - its program alone; the window says why for the latter. -- Guests' broadcasts are not in the layout yet, so only sources local to the - studio play as layers. `priority` and `toggle` are not applied. +`moq-studio --watch https://relay.example/anon --broadcast my-show` plays a +studio's composition in a window as a viewer sees it: the program at +`my-show`, with each picture over it and each sound mixed with it. The URL +names the relay as `--connect` does, and must reach every broadcast the +composition lists, guests' included; `--connect-token` and the `--quic-*` +flags apply to it, and `--connect`, `--failover`, and `--room` are refused +beside it. It is a prototype for settling the composition's format, which +goes once `moq play` plays compositions. + +- Each composition takes effect when the program plays the frame it is + stamped with, and its pictures' keyframes play from its `start`. +- Each source plays its own broadcast on the program's clock, its timestamps + moved by its `offset`: a picture shows when the program's picture stamped + alike does, and a sound plays with the program's sound stamped alike, at + its `gain`, `muted`, `pan`, and `delay`. A frame too late for its moment is + dropped. A new offset plays on without fetching the broadcast again. A + source's sound is fetched only while the composition gives it a `sound`, + and gaining or losing one does not fetch its picture again. +- A source whose `instance` another stream replaced shows nothing until a + composition names the new one, and a composition naming another program + stream waits for it, so neither plays on a timeline its offset is not for. +- A broadcast without a composition, or with one of another `version` or one + it cannot place, such as one listing a broadcast above the relay URL's + path, plays its program alone; the window says why for the latter. +- `priority` and `toggle` are not applied. - Sound needs a build with `capture`; without it the window says so in the log and plays the pictures alone. diff --git a/justfile b/justfile index bf8397ad..01586792 100644 --- a/justfile +++ b/justfile @@ -58,7 +58,7 @@ render-sw: icd=("$mesa"/share/vulkan/icd.d/lvp_icd.*.json) export VK_DRIVER_FILES="${icd[0]}" cargo nextest run --locked -p moq-scene --lib --run-ignored only -E 'test(/^compositor::tests::/) | test(/^browser::layers::tests::/)' - cargo nextest run --locked -p moq-studio --features openh264 --run-ignored only -E '(test(/^engine::tests::/) - test(=engine::tests::the_virtual_camera_reaches_a_pipewire_app)) | test(=setup::tests::times_every_canvas)' + cargo nextest run --locked -p moq-studio --features openh264 --run-ignored only -E '(test(/^engine::tests::/) - test(=engine::tests::the_virtual_camera_reaches_a_pipewire_app)) | test(=setup::tests::times_every_canvas) | test(=watch::tests::plays_a_guest_where_the_studio_did)' cargo nextest run --locked -p moq-studio --features cef --run-ignored only -E 'binary(browser)' sh/pipewire-session.sh cargo nextest run --locked -p moq-studio --features pipewire --run-ignored only -E 'test(=engine::tests::the_virtual_camera_reaches_a_pipewire_app)' diff --git a/quest/m0/README.md b/quest/m0/README.md index 6cab862c..442abe8f 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -88,7 +88,7 @@ each encoding costs an encoder session and consumer GPUs have few, so the [viewer-side composition](/quest/m2/compose/README.md) epic left m0 (now m2, replanned 2026-11-08 around independently published sources layered over the program). Overlays and animation render into the program here; that epic's -layout track carries only the layers of independently published sources. +composition lists only the independently published sources over it. Changed 2026-11-09 with the maintainer (/quest-plan): the studio gets one designed layout, a live picture with pages around it, in place of sixteen diff --git a/quest/m2/compose/README.md b/quest/m2/compose/README.md index 86424297..1ddafdb8 100644 --- a/quest/m2/compose/README.md +++ b/quest/m2/compose/README.md @@ -57,7 +57,9 @@ offset onto the composition's clock, and every independently published source, local or guest, its own broadcast (rejected: a layer layout track in the program's broadcast naming tracks; local sources as tracks of the program's broadcast, the 2026-10-26 rule this reverses). The studio -prototypes the format first. See [the composition](/quest/m2/compose/layout.md). +prototypes the format first: `rs/moq-scene/src/composition.rs`, documented in +the README's "Composition". [Moving it into its broadcast's +catalog](/quest/m2/compose/layout.md) waits for hang's timed catalogs. moq's side, in its own quest tree (m3): hang specifies the composition, and `moq play` and `@moq/watch` composite it. moq's three layers quests were diff --git a/quest/m2/compose/controls.md b/quest/m2/compose/controls.md index c3b2390c..c355b6f4 100644 --- a/quest/m2/compose/controls.md +++ b/quest/m2/compose/controls.md @@ -2,10 +2,10 @@ ## Goal -A viewer watching a studio broadcast can switch off any layer the layout -marks toggleable (a facecam, a screen share), and stops fetching its track. -The layout's per-layer priorities steer delivery, so under congestion the -least important sources degrade first. The creator sets both per source in +A viewer watching a studio broadcast can switch off any source the +composition marks toggleable (a facecam, a screen share), and stops fetching +its broadcast. The composition's per-source priorities steer delivery, so +under congestion the least important sources degrade first. The creator sets both per source in the studio. ## Plan @@ -26,5 +26,6 @@ Map layer priority onto the subscription priorities moq-lite already has (`moq/doc/concept/moq-lite.md`), rather than a second mechanism. A toggled-off layer is unsubscribed, so it costs the viewer and the relay nothing. The studio keeps encoding it, so toggling it back on can reach its recent past. -Subscriptions are per track, not per layer: a source shown by several layers -stays subscribed while any of them is on, at the highest priority among them. +The composition carries `priority` and `toggle` on each source's entry +(`rs/moq-scene/src/composition.rs`), not on each picture, so a source shown by +several pictures is one subscription, on or off as a whole. diff --git a/quest/m2/compose/hang-layers.md b/quest/m2/compose/hang-layers.md index f3b2b43b..c64d13bf 100644 --- a/quest/m2/compose/hang-layers.md +++ b/quest/m2/compose/hang-layers.md @@ -19,8 +19,8 @@ quests (rejected: "both quests", unnamed, which left out `@moq/watch`). Decided 2026-11-12 (maintainer, /quest-unblock): hang specifies a composition, a broadcast listing a catalog per source, rather than a layer -layout track; see [the composition](/quest/m2/compose/layout.md), whose -prototype moq's spec starts from. moq's three quests below were planned for a +layout track. The studio's prototype, which moq's spec starts from, is +`rs/moq-scene/src/composition.rs`, documented in the README's "Composition". moq's three quests below were planned for a layer layout track and are replanned in moq's tree. ## Waiting @@ -39,6 +39,5 @@ layer layout track and are replanned in moq's tree. ## Required -- [The studio publishes a composition](/quest/m2/compose/layout.md) - the prototype hang's spec starts from - [Viewer controls](/quest/m2/compose/controls.md) - toggles and priorities the spec carries - [The studio is caught up to moq's main](/quest/m0/pin-main.md) - moq's breaking changes since moq-dev/moq#8272, so this pin takes only its own diff --git a/quest/m2/compose/layout.md b/quest/m2/compose/layout.md index 830d3bd5..764ab8cf 100644 --- a/quest/m2/compose/layout.md +++ b/quest/m2/compose/layout.md @@ -1,53 +1,41 @@ -# [L] The studio publishes a composition +# [M] The composition is in its broadcast's hang catalog ## Goal -The studio publishes a composition: a broadcast of its own whose catalog -lists other broadcasts, the program and each independently published source, -local or guest, with where and when each shows and how its sound mixes. Each -listed broadcast keeps its own timeline, and the composition carries the -offset that puts it on the composition's media clock. The `moq-studio --watch` -prototype (`rs/moq-studio/src/watch.rs`) plays a composition, guests -included, and it replaces the provisional `live.kixel.studio.layers` track -(`rs/moq-scene/src/layout.rs`, and the README's "Layer layout"). +The composition the studio publishes (`rs/moq-scene/src/composition.rs`, the +README's "Composition") travels in the composition broadcast's hang catalog, +each update applied at its timestamp on the program's clock, in place of the +interim `live.kixel.studio.composition` JSON snapshot track. `moq-studio +--watch` (`rs/moq-studio/src/watch.rs`) reads it from that catalog, and +`README.md`'s "Composition" and "Watching" say so. ## Plan -Decided 2026-11-12 (maintainer, /quest-unblock), settling how the layout -names a guest: - -- **Naming:** ✅ a composition, a catalog of catalogs (rejected: the layer - layout naming each guest's broadcast with an offset; moq relaxing hang's - `include` for broadcasts on their own timeline; republishing guests' - tracks in the studio's broadcast). hang's `include` - ([moq-dev/moq#7943](https://github.com/moq-dev/moq/pull/7943)) requires an - included broadcast to share the listing catalog's PTS timeline, - `media.clock`, and `media.join.interval`, which a guest's does not; a - composition keeps `include`'s meaning and says how each timeline maps. -- **Home:** ✅ its own broadcast (rejected: a section of the program's - catalog, which is `include` with a different clock). The program is one of - its children, and a player that knows no compositions still plays the - program. -- **Local sources:** ✅ every independently published source is its own - broadcast (rejected: layers naming a broadcast and track, with local - sources staying tracks of the program's broadcast at offset zero). This - reverses the 2026-10-26 rule that local sources are tracks in the studio's - one broadcast, so the composition lists only catalogs. -- **Format:** ✅ the studio prototypes it, provisional and versioned, before - moq's hang specifies it (rejected: straight to hang's spec), as the - 2026-11-08 layer layout did. - -Recommended, not decided: the composition's media clock is the program's, so -the program's offset is zero; each child's entry carries what the provisional -layer format carries per layer today (placement, crop, opacity, visibility, -keyframes, toggleability, priority, and gain, mute, pan, and delay for -sound); and a change is a new snapshot group stamped on the composition's -clock, as the layers track is now. - -`Source::publishes` refuses a broadcast source today, since it has no track -in the studio's broadcast; under a composition a guest needs no track of the -studio's. `README.md`'s "Published independently" and "Layer layout" become -the composition's. - -Bump the format's `version`, since naming broadcasts instead of tracks is not -additive. +Decided 2026-11-12 (maintainer, /quest-complete on +[moq-dev/studio#146](https://github.com/moq-dev/studio/pull/146)), which +landed the composition on the interim track: + +- **Home:** ✅ the composition broadcast's hang catalog, applied at each + update's timestamp (rejected: shipping the interim JSON snapshot track as + the format). moq plans timed catalogs in + [catalog-at-timestamp](https://github.com/moq-dev/moq/blob/main/quest/m2/catalog-at-timestamp.md) + (m2, on lite-07's time addressing), and the rest of its composition plan in + [composed broadcasts](https://github.com/moq-dev/moq/blob/main/quest/m3/layers/README.md). +- **Shape:** ✅ kept as #146 has it: each child entry carries `sound`, + `priority`, and `toggle`, and placements live in `pictures`, each naming a + child, so one broadcast can show in several places (rejected: one + placement per child, as moq's spec quest said; moq's + `quest/m3/layers/spec.md` is updated to match). + +Recommended, not decided: the composition goes in a section of the catalog +under the studio's own name until hang specifies one, with `version` kept, +since moq's spec may still change it; the catalog also lists the program's +renditions, as moq's plan has a composition list its main's, so a player +that knows no compositions plays the program from it. A child's offset and +each placement then change with one timed catalog update, and the watcher +applies each when the program plays the frame stamped alike, as it applies +the track's groups now. + +## Required + +- [hang's timed catalog is pinned](/quest/m2/compose/pin-timed-catalog.md) - hands the studio each catalog update with its timestamp diff --git a/quest/m2/compose/pin-timed-catalog.md b/quest/m2/compose/pin-timed-catalog.md new file mode 100644 index 00000000..af87a3dd --- /dev/null +++ b/quest/m2/compose/pin-timed-catalog.md @@ -0,0 +1,29 @@ +# [S] hang's timed catalog is pinned + +## Goal + +The `moq` submodule includes hang applying each catalog update at its +timestamp, from moq's +[hang applies catalog updates at their timestamp](https://github.com/moq-dev/moq/blob/main/quest/m2/catalog-at-timestamp.md), +so `rs/hang`'s catalog consumer hands the studio each update with its +timestamp, which [the composition](/quest/m2/compose/layout.md) needs to move +into its broadcast's catalog. + +## Plan + +Once the wait clears, advance it with `just upgrade-moq` in its own PR, +fixing whatever the new API breaks here. That quest builds on moq's lite-07 +time addressing (groups addressed by their start), so the pin may cross +that model change too, which is why it is sized above the usual pin. + +## Waiting + +- moq-dev/moq: the catalog-at-timestamp quest lands. Check: its PR has + merged on moq's `main`, deleting `quest/m2/catalog-at-timestamp.md`, and + `rs/hang/src/catalog/consumer.rs` returns each catalog with its timestamp. + A quest file that moved, or went away without its work landing, does not + count. + +## Required + +- [The studio is caught up to moq's main](/quest/m0/pin-main.md) - moq's breaking changes since moq-dev/moq#8272, so this pin takes only its own diff --git a/quest/m2/compose/server.md b/quest/m2/compose/server.md index 5dffb041..4117f5ce 100644 --- a/quest/m2/compose/server.md +++ b/quest/m2/compose/server.md @@ -27,14 +27,10 @@ too and moq never depends on this repository. Settle its name in this quest and document it in `README.md`. It flattens the composition the studio publishes -([the composition](/quest/m2/compose/layout.md)); until that lands, it -flattens the provisional `live.kixel.studio.layers` track (`rs/moq-scene/src/layout.rs`, -and the README's "Layer layout"). +(`rs/moq-scene/src/composition.rs`, and the README's "Composition"), placing +each listed broadcast on the program's clock at its offset as the +`moq-studio --watch` prototype (`rs/moq-studio/src/watch.rs`) does. ## Required - [Destinations](/quest/m0/destinations.md) - the egress library the server pushes RTMP and WHIP through - -## Related - -- [The studio publishes a composition](/quest/m2/compose/layout.md) - the format it flattens diff --git a/rs/moq-scene/Cargo.toml b/rs/moq-scene/Cargo.toml index 999c472c..90914941 100644 --- a/rs/moq-scene/Cargo.toml +++ b/rs/moq-scene/Cargo.toml @@ -99,7 +99,7 @@ x11rb = { version = "0.13", features = ["dri3"] } wayland-client = "0.31" [dev-dependencies] -# Reads the layer layout track back as a player would. +# Reads the composition track back as a player would. moq-json = { workspace = true, features = ["async"] } # An encoder on every machine, for the source track and replay tests. moq-video = { workspace = true, features = ["openh264"] } diff --git a/rs/moq-scene/src/collection.rs b/rs/moq-scene/src/collection.rs index 56784c7d..328966ba 100644 --- a/rs/moq-scene/src/collection.rs +++ b/rs/moq-scene/src/collection.rs @@ -40,13 +40,15 @@ pub struct Collection { /// wherever the source is placed. A source without an entry is unfiltered. #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] pub video: BTreeMap, - /// The sources published independently, by name: each one's own track - /// goes to the relay beside the program, and the - /// [layer layout](crate::layout) says where viewers show it over the + /// The sources published independently, by name: each reaches viewers + /// as a broadcast of its own, a local source's track in one the studio + /// publishes and a broadcast source as its publisher's, and the + /// [composition](crate::composition) says where viewers show it over the /// program and how they play its sound. The program and every audio - /// scene's mix leave it out. Only a source with a track of its own can - /// be, and not one with its background removed, since its track would - /// show the background. Wherever it is placed, it takes only what every + /// scene's encoded mix leave it out, but the mix people in the studio + /// hear holds it ([`Sources::listened`](crate::source::Sources::listened)). Only a broadcast or a source with a track of + /// its own can be, and not one with its background removed, since its + /// track would show the background. Wherever it is placed, it takes only what every /// player can reproduce, and no scene that another scene draws holds it. #[serde(default, skip_serializing_if = "BTreeSet::is_empty")] pub published: BTreeSet, @@ -469,11 +471,11 @@ impl Collection { "`{name}` draws scene `{scene}`, which is not a scene" ))); } - if let Source::Layer { broadcast, .. } = source - && !matches!(self.sources.get(broadcast), Some(Source::Broadcast { .. })) + if let Source::Layer { clock, .. } = source + && !matches!(self.sources.get(clock), Some(Source::Broadcast { .. })) { return Err(Error::Invalid(format!( - "`{name}` plays on `{broadcast}`, which is not a broadcast source" + "`{name}` plays on `{clock}`, which is not a broadcast source" ))); } let Source::Clone { source } = source else { continue }; @@ -518,10 +520,17 @@ impl Collection { "`{name}` is published, but there is no such source" ))); }; - if !source.publishes() { + if !source.independent() { return Err(Error::Invalid(format!( - "`{name}` cannot be published independently: only a camera, display, window, microphone, \ - system audio, color, image, media, or replay source has a track of its own" + "`{name}` cannot be published independently: only a broadcast, or a camera, display, window, \ + microphone, system audio, color, image, media, or replay source with a track of its own, can be" + ))); + } + let segment = !name.contains('/') && moq_net::Path::new(name).is_ok() && name != "." && name != ".."; + if source.publishes() && !segment { + return Err(Error::Invalid(format!( + "`{name}` is published independently, so its name is one segment of its broadcast's path: no \ + `/`, not `.` or `..`, and a `\\` only before another" ))); } if self @@ -638,9 +647,9 @@ impl Collection { } /// Refuse an item showing a source published independently that the - /// layer layout cannot carry: one blended or filtered past opacity, which + /// composition cannot carry: one blended or filtered past opacity, which /// not every player can reproduce, or one in a scene another scene draws, - /// which the layout cannot place. + /// which the composition cannot place. fn validate_published(&self) -> Result<(), Error> { let nested: BTreeSet<&str> = self .sources @@ -1254,7 +1263,7 @@ fn is_true(value: &bool) -> bool { /// What a source or item is drawn through, in order. /// /// Every filter is a pure function of the picture, so every host that composes -/// a scene with moq-scene draws it the same way, and it travels in a layout as +/// a scene with moq-scene draws it the same way, and it travels in a composition as /// its name and settings. The CSS ones mean what the CSS Filter Effects /// function of the same name does, in the gamma-encoded sRGB the canvas holds, /// with lengths in canvas pixels. A filter naming another source uses that @@ -1855,18 +1864,23 @@ pub enum Source { /// same frame. A scene drawing it cannot go on program, and the top layer /// cannot draw it, since either would draw the program into itself. Program, - /// A source a studio broadcast publishes independently, played from the - /// broadcast the [`Source::Broadcast`] called `broadcast` plays, on that - /// source's playout clock, so it lands with the program as the - /// [layer layout](crate::layout) places it. Never in a scene file: only - /// a player of the layout makes one. + /// A broadcast a [composition](crate::composition) lists, played on the + /// playout clock of the [`Source::Broadcast`] called `clock`, its + /// timestamps moved `offset` onto that clock's, so it lands with the + /// program as the composition places it. Never in a scene file: only a + /// player of the composition makes one. #[serde(skip)] Layer { - /// The broadcast source whose broadcast and clock it plays on. - broadcast: String, - /// Its entry under the broadcast catalog's - /// [`SECTION`](crate::publish::SECTION) `sources`. - source: String, + /// The broadcast source whose clock it plays on: the program. + clock: String, + /// Its broadcast's path. + path: String, + /// The stream `offset` is for, or `None` for whichever serves the + /// path: another stream shows nothing, unless a relay minted it. + instance: Option, + /// Microseconds added to its timestamps to put them on `clock`'s + /// timeline. A change plays on without opening the broadcast again. + offset: i64, }, } diff --git a/rs/moq-scene/src/composition.rs b/rs/moq-scene/src/composition.rs new file mode 100644 index 00000000..75597804 --- /dev/null +++ b/rs/moq-scene/src/composition.rs @@ -0,0 +1,621 @@ +//! The composition: a broadcast of the studio's own whose catalog lists other +//! broadcasts, the program and each source published independently, local or +//! a guest's, with where and when each shows and how its sound mixes, so a +//! player places each as the studio would have. +//! +//! The program leaves a published source out, so a player draws each +//! [picture](Composition::pictures) over the program, back to front, and mixes +//! each [sound](Child::sound) with the program's audio. A composition carries +//! only what every player can reproduce: a picture's place, crop, opacity, +//! visibility, and keyframes, and a sound's gain, mute, pan, and delay. +//! [`Collection::validate`] refuses a published source placed with anything +//! more, such as a blur or a blend mode. +//! +//! Each listed broadcast keeps its own timeline. The composition's media clock +//! is the program's, and each [`Child`] carries the offset that puts its +//! broadcast's timestamps on it: zero for a local source, which the studio +//! stamps on the program's clock, and for a guest, how far its stamps sat from +//! the program's when the studio played them. +//! +//! It travels as JSON on [`TRACK`] in snapshot mode, in a broadcast of its own +//! [beneath the program's](PATH), so a viewer joining late reads the +//! composition in force. The format is the studio's own, provisional until +//! hang specifies a composition. + +use std::collections::BTreeMap; +use std::time::Duration; + +use serde::{Deserialize, Serialize}; +use serde_with::{DisplayFromStr, DurationMicroSeconds, serde_as}; + +use crate::Error; +use crate::collection::{Audio, Collection, Item, Layer, Size}; + +/// The composition broadcast's track carrying the composition, listed in its +/// catalog's `json` section. +pub const TRACK: &str = "live.kixel.studio.composition"; + +/// Where the composition's broadcast is, beneath the program's. +pub const PATH: &str = "composition"; + +/// Where each local source published independently has a broadcast of its +/// own, beneath the program's, under its name: `/source/`. +pub const SOURCES: &str = "source"; + +/// How far a child's offset may wander before a new composition carries it. +/// +/// The offset is measured from a playout clock that re-anchors on every write +/// of a guest's sound, so it jitters by a millisecond or so while the guest +/// plays steadily. Well under the 45 ms of sound leading the picture a viewer +/// can detect (ITU-R BT.1359), it keeps a viewer's guest in step with the +/// studio's without a group per frame. +pub const DRIFT: Duration = Duration::from_millis(10); + +/// Where and when each broadcast the composition lists shows or plays. +#[serde_as] +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct Composition { + /// The format's version, first so a reader of another one fails on it. + pub version: Version, + /// The program's canvas, which pictures are placed on in pixels. + pub canvas: Size, + /// When the scene on program went on, on the composition's media clock: + /// the moment its pictures' keyframes play from. In microseconds on the + /// wire. + #[serde_as(as = "DurationMicroSeconds")] + pub start: Duration, + /// The program, whose clock the composition's is. + pub program: Program, + /// Each source published independently that a picture shows or the + /// program's audio plays, by source name. + #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] + pub sources: BTreeMap, + /// Each picture over the program, back to front: the scene on program's, + /// then the top layer's. Each is a scene file item whose `source` names a + /// key of [`sources`](Self::sources), whose filters are only opacity, and + /// whose keyframes play from [`start`](Self::start). + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub pictures: Vec, +} + +impl Composition { + /// The composition of `collection` with the scene called `scene` on + /// program since `start` on the media clock, or no scene for `None`, + /// drawn over `program`. Each source published independently is listed + /// as `children` locates it, its sound filled in; one `children` leaves + /// out, such as a guest offline, is left out with its pictures, as the + /// studio shows nothing of it either. + pub fn new( + collection: &Collection, + scene: Option<&str>, + start: Duration, + program: Program, + mut children: BTreeMap, + ) -> Self { + let pictures: Vec = pictures(collection, scene, true) + .into_iter() + .filter(|item| children.contains_key(&item.source)) + .collect(); + let heard = collection.program_audio().map(|audio| collection.heard(audio)); + for source in heard.into_iter().flatten() { + if let Some(child) = children.get_mut(source) { + child.sound = Some(collection.audio.get(source).cloned().unwrap_or_default()); + } + } + children.retain(|name, child| child.sound.is_some() || pictures.iter().any(|item| item.source == *name)); + Self { + version: Version, + canvas: Size { + width: collection.canvas.width, + height: collection.canvas.height, + }, + start, + program, + sources: children, + pictures, + } + } +} + +/// Each item of the scene called `scene`, then of the top layer if `top`, +/// showing a source published independently, back to front, shown or hidden: +/// its `source` names the published source, and its crop and filters fold in +/// its source's, as the compositor draws them. +pub(crate) fn pictures(collection: &Collection, scene: Option<&str>, top: bool) -> Vec { + let scene = scene.filter(|scene| collection.scene(scene).is_some()); + let places = scene.map(Layer::scene).into_iter().chain(top.then(Layer::top)); + let mut pictures = Vec::new(); + for place in places { + for item in collection.items(&place).unwrap_or_default() { + let source = collection.capture(&item.source); + if !collection.published.contains(source) { + continue; + } + pictures.push(Item { + source: source.to_string(), + crop: collection.crop(item), + filters: collection.filters(item).cloned().collect(), + ..item.clone() + }); + } + } + pictures +} + +/// The format's version: deserializing refuses any but +/// [`Version::CURRENT`]. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(into = "u32", try_from = "u32")] +pub struct Version; + +impl Version { + /// The version this build reads and writes. Version 1 was the layer + /// layout, which named tracks of the program's broadcast. + pub const CURRENT: u32 = 2; +} + +impl From for u32 { + fn from(_: Version) -> Self { + Version::CURRENT + } +} + +impl TryFrom for Version { + type Error = String; + + fn try_from(found: u32) -> Result { + match found { + Self::CURRENT => Ok(Self), + found => Err(format!( + "composition version {found} is not supported; this build reads version {}", + Self::CURRENT + )), + } + } +} + +/// The program's broadcast. +#[serde_as] +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct Program { + /// Its path, relative to the composition's, as hang's references are. + pub broadcast: moq_net::path::Relative, + /// The stream whose timeline the composition's clock is, so a player + /// waits for it rather than placing anything on another's. + #[serde(default, skip_serializing_if = "Option::is_none")] + #[serde_as(as = "Option")] + pub instance: Option, +} + +/// One broadcast the composition places over the program or mixes with it. +#[serde_as] +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct Child { + /// Its path, relative to the composition's, as hang's references are. + pub broadcast: moq_net::path::Relative, + /// The stream [`offset`](Self::offset) holds for, or `None` for whichever + /// serves the path, as for a source the studio stamps on the program's + /// clock. A relay-minted instance is never named, since it cannot match + /// another relay's. + #[serde(default, skip_serializing_if = "Option::is_none")] + #[serde_as(as = "Option")] + pub instance: Option, + /// Microseconds added to its timestamps to put them on the composition's + /// clock, negative for a broadcast stamped ahead of it. + #[serde(default, skip_serializing_if = "is_zero")] + pub offset: i64, + /// Its gain, mute, pan, and delay in the program's audio, or `None` when + /// the program's audio does not play it. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub sound: Option