Repository navigation
feat(hang): VR projection, stereo, and ambisonic catalog metadata - #5543
Conversation
|
Thanks for the VR metadata work! Where it stands:
Ping when it's rebased and green and we'll review. (Written by Claude Opus 5.5) |
Move projection to each video rendition beside color, add stereo (mono/left-right/top-bottom) in Rust and JS, and add the audio layout in Rust. Unknown values are preserved and treated as absent; unknown fields on a known projection are ignored. Restore the cross-kind duplicate rendition check and the color tests the earlier merge dropped. Document the fields in the hang draft and doc/concept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 13 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Thanks @nuts-rice for starting this. Spatial metadata is a great addition, and your
The description lists each decision with its alternatives. Your two TODOs, (Written by Claude Opus 5.5) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: eadbcbc747
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| }), | ||
| ), | ||
| }), | ||
| z.transform((raw) => ({ kind: "unknown" as const, raw })), |
There was a problem hiding this comment.
Keep unknown projections in the wire shape
When ProjectionSchema sees a future projection such as { kind: "cubemap", face: "front" }, this transform returns { kind: "unknown", raw: ... } rather than the original wire object. A JS relay that republishes a parsed catalog passes it through Json.Snapshot.Encoder.update (js/json/src/snapshot/encoder.ts:201-205), which parses and stringifies that wrapper; parsing the republished catalog then wraps it again. Rust's UnknownProjection instead serializes the original object, so the implementations do not share the promised round-trip behavior. Keep the unknown output wire-shaped or unwrap raw during encoding.
AGENTS.md reference: AGENTS.md:L111-L111
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Declining for this PR. ProjectionSchema deliberately copies UnknownContainerSchema in js/hang/src/catalog/container.ts, which has had the same { kind: "unknown", raw } shape (and the same Rust/JS asymmetry) since unknown containers were added. The literal kind keeps kind === "equirectangular" narrowing, and raw is the documented way for a republisher to write the original back. No JS code reparses and republishes a catalog today: CatalogProducer builds its catalog from scratch and passes no schema to the encoder.
If we want a parsed JS catalog to stringify back to its wire shape, the fix belongs in both schemas at once, so I'll suggest it as a follow-up rather than diverge the two here.
(Written by Claude Opus 5.5)
| Rectilinear, | ||
| Equirectangular { | ||
| #[serde(default)] | ||
| fov: Option<f64>, |
There was a problem hiding this comment.
Reject explicit null spatial metadata in Rust
When a full catalog snapshot contains "fov": null, this Option<f64> decodes it as None, so Projection::check accepts the malformed known projection; the same Option behavior accepts top-level projection, stereo, and layout nulls as absent. The JS z.optional schemas reject these values, and the draft permits omission or the declared object/string/number types, not null. Use presence-aware deserialization so explicit null is rejected while a missing field still defaults.
AGENTS.md reference: AGENTS.md:L17-L17
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Declining for this PR. Every optional field in the Rust catalog is an Option with #[serde(default)], so "color": null, "bitrate": null, "label": null and the rest already decode as absent. Making only the new fields presence-aware would be inconsistent, and null-as-absent lands on the safe default (flat, mono, codec channel order), so it can't make a player misrender. A stricter null policy would be a catalog-wide change in its own PR; I'll list it as a possible follow-up.
(Written by Claude Opus 5.5)
|
Merge summary for
(Written by Claude Opus 5.5) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ed118c4459
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| /// | ||
| /// Live state, like [`color`](Self::color). | ||
| #[serde(default)] | ||
| pub projection: Option<Projection>, |
There was a problem hiding this comment.
Preserve spatial fields on transcoded ladder rungs
When moq-transcode builds generated ladder rungs from a rendition with projection: equirectangular or a packed stereo, rung_entry in rs/moq-transcode/src/catalog.rs:326-340 starts from the encoder probe and inherit at lines 346-350 copies only enabled and preference; neither transfers these fields, while the rung pipeline only scales the complete frame. Every generated rung therefore advertises the absent defaults rectilinear and mono, so VR clients render the transcoded output incorrectly. Copy the source projection and stereo into each rung unless a pixel transform changes them.
Useful? React with 👍 / 👎.
| impl AudioLayout { | ||
| /// Refuse a channel count this layout cannot carry. | ||
| pub(crate) fn check(&self, channel_count: u32) -> crate::Result<()> { | ||
| if *self == Self::Ambisonic { |
There was a problem hiding this comment.
Validate Unknown layouts by their emitted name
When a programmatic publisher sets AudioLayout::Unknown("ambisonic".into()) with six channels, this variant equality skips the square-count check even though DisplayFromStr emits the recognized wire value "ambisonic". Outputs::emit in rs/moq-mux/src/catalog/producer.rs:212-214 consequently publishes a catalog that both Rust and JS consumers reject. Since Unknown(String) is publicly constructible, validate by the emitted name or prevent reserved names from entering that variant.
AGENTS.md reference: rs/AGENTS.md:L42-L43
Useful? React with 👍 / 👎.
Every estimate a container producer published read its rendition's config through a full catalog snapshot, three times per change, so each frame cloned every rendition on the broadcast. VR metadata (#5543) made each VideoConfig clone larger, which is the +3.3% the m0 audit bisected to it. Clone only the entry under the catalog lock, once per estimate. Callgrind, one iteration of catalog_estimate_fanout (instructions): - hang/tracks_1/1: 19,149,482 -> 12,068,527 (-37.0%) - hang/tracks_4/4: 106,157,457 -> 59,002,822 (-44.4%) - hang/tracks_16/16: 1,074,313,398 -> 509,137,148 (-52.6%) - msf_references/tracks_4/4: 110,861,752 -> 58,856,411 (-46.9%) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Started by @nuts-rice; finished on top of their commits at the maintainer's request.
Problem
A hang catalog can't say that a video is 360/VR180, that its frames pack two eyes, or that an audio track is ambisonic, so a headset player has to guess.
Approach
Three optional catalog fields, mirrored in
rs/hangandjs/hang:projection:{ "kind": "rectilinear" }(the default) or{ "kind": "equirectangular", "fov": 180 }.fovis the degrees of longitude the picture spans, centred straight ahead, in (0, 360], defaulting to 360. Latitude always spans the full 180 degrees, so VR180 isfov: 180.stereo:"mono"(the default),"left-right", or"top-bottom". The left eye is in the left or top half.layout:"ambisonic"means full-sphere AmbiX (ACN order, SN3D normalization). The ambisonic order comes fromnumberOfChannels, which must be a square of at least 4.Forward compatibility and validation:
kind,stereovalue, orlayoutvalue is kept verbatim so a republish round-trips it (in JS, an unknown projection parses as{ kind: "unknown", raw }, like an unknown container). A consumer treats it as absent, which is also what a reader that predates these fields does.kind, afovoutside (0, 360] (NaN and infinity included), a non-stringstereoorlayout, orambisonicwith a channel count that isn't a square of at least 4. JS refuses these while parsing. Rust refuses the projection while parsing and the ambisonic channel count incheck_renditions, which every Rust catalog consumer and producer already calls.version: a reader that ignores them shows a flat, mono picture and plays the channels in codec order. That is a worse result, not a wrong one.Also fixed: the earlier merge of
mainhad dropped the cross-kind duplicate rendition check incheck_renditionsand the videocolorJS tests. That is whyTestfailed. Both are back.Decisions
The maintainer settled 1 to 4 and 6 (✅ marks the choice); 5 follows from placing the fields next to
color. The alternatives are kept for the record.color. A transcode can reproject or drop an eye, the same way an HDR to SDR transcode changescolor. A player can then pick the rendition it can render, and audiolayoutalready sits on the rendition.rotation/flip(the original shape). This can't describe a flat fallback rendition of a 360 source.stereovaluesleft-right/top-bottom, matching WebXRXRLayerLayout, Google Spherical Video V2, and YouTube. The name says where each eye goes.side-by-side/top-bottom(the original names). These leave the eye order unclear.fovis shapedequirectangular, with latitude always 180. That covers 360 and VR180.XREquirectLayer). Cropped poles are rare, and a vertical span can be added later without breaking anything, since readers ignore unknown fields.fov, plus a separatehalf-equirectangularkind (Apple). Less general.color, because a publisher may only learn these values from the bitstream (st3d/sv3d, SEI).layout: "ambisonic", meaning full-sphere AmbiX (ACN order, SN3D normalization), with the order taken fromnumberOfChannels.Impact
projectionandstereoand optional audio renditionlayout. Nothing is removed, andversiondoesn't change.hang(Rust), additive only:VideoConfig::projection,VideoConfig::stereo,AudioConfig::layout.Projection,UnknownProjection,Stereo,AudioLayout.Errorvariants:InvalidFieldOfView,InvalidAmbisonicChannels.Errorare#[non_exhaustive], so none of this breaks callers.@moq/hang(JS), additive only: newProjectionSchema/Projectionexports, plus the new fields onVideoConfigandAudioConfig.drafts/draft-lcurley-moq-hang.mddocuments the fields, with a moq-hang-04 changelog entry.doc/concept/hang.mdgets a short "Spatial media" entry.new()initializers, a few lines incheck_renditions, and a small hunk inaudio.ts.Checks
just checkpasses. One run hitredundant_importers_switch_through_bufferbloat_on_lite_07, a known wall-clock flake tracked inquest/m1/test-flakes-2/redundant-failover-drift.md. It passed on rerun, and a full rerun was clean.just drafts checkandbun run --cwd doc checkpass.Follow-ups
Suggested; not in this PR:
<moq-watch>renders these fields through WebXR layers (XREquirectLayer, withlayoutmapped fromstereo). This replaces the original TODO about an end-to-end path in<moq-watch>.containerandprojectionvalues stringify back to their wire shape, so a JS republisher round-trips them like Rust does (review thread).nullon optional fields, catalog-wide (review thread).moq-muximporters fill the fields from MP4st3d/sv3d, MatroskaProjection/StereoMode, and Opus channel mapping family 2.🤖 Generated with Claude Code
(Written by Claude Opus 5.5)