Skip to content

feat(hang): VR projection, stereo, and ambisonic catalog metadata - #5543

Merged
kixelated merged 10 commits into
moq-dev:mainfrom
nuts-rice:feat/hang-vr-metadata
Oct 21, 2026
Merged

kixelated merged 10 commits into
moq-dev:mainfrom
nuts-rice:feat/hang-vr-metadata

Conversation

@nuts-rice

@nuts-rice nuts-rice commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Started by @nuts-rice; finished on top of their commits at the maintainer's request.

Problem

A hang catalog can't say that a video is 360/VR180, that its frames pack two eyes, or that an audio track is ambisonic, so a headset player has to guess.

Approach

Three optional catalog fields, mirrored in rs/hang and js/hang:

  • Video rendition projection: { "kind": "rectilinear" } (the default) or { "kind": "equirectangular", "fov": 180 }. fov is the degrees of longitude the picture spans, centred straight ahead, in (0, 360], defaulting to 360. Latitude always spans the full 180 degrees, so VR180 is fov: 180.
  • Video rendition stereo: "mono" (the default), "left-right", or "top-bottom". The left eye is in the left or top half.
  • Audio rendition layout: "ambisonic" means full-sphere AmbiX (ACN order, SN3D normalization). The ambisonic order comes from numberOfChannels, which must be a square of at least 4.

Forward compatibility and validation:

  • An unknown projection kind, stereo value, or layout value is kept verbatim so a republish round-trips it (in JS, an unknown projection parses as { kind: "unknown", raw }, like an unknown container). A consumer treats it as absent, which is also what a reader that predates these fields does.
  • Readers ignore unknown fields on a known projection kind, so a parameter added later doesn't break older readers.
  • Malformed values are refused: a projection without a string kind, a fov outside (0, 360] (NaN and infinity included), a non-string stereo or layout, or ambisonic with a channel count that isn't a square of at least 4. JS refuses these while parsing. Rust refuses the projection while parsing and the ambisonic channel count in check_renditions, which every Rust catalog consumer and producer already calls.
  • None of the fields change version: a reader that ignores them shows a flat, mono picture and plays the channels in codec order. That is a worse result, not a wrong one.

Also fixed: the earlier merge of main had dropped the cross-kind duplicate rendition check in check_renditions and the video color JS tests. That is why Test failed. Both are back.

Decisions

The maintainer settled 1 to 4 and 6 (✅ marks the choice); 5 follows from placing the fields next to color. The alternatives are kept for the record.

  1. Where the video fields live
    • ✅ On each rendition, next to color. A transcode can reproject or drop an eye, the same way an HDR to SDR transcode changes color. A player can then pick the rendition it can render, and audio layout already sits on the rendition.
    • On the video section, next to rotation/flip (the original shape). This can't describe a flat fallback rendition of a 360 source.
  2. Names for the stereo values
    • ✅ left-right / top-bottom, matching WebXR XRLayerLayout, Google Spherical Video V2, and YouTube. The name says where each eye goes.
    • side-by-side / top-bottom (the original names). These leave the eye order unclear.
  3. How fov is shaped
    • ✅ One horizontal span on equirectangular, with latitude always 180. That covers 360 and VR180.
    • Separate horizontal and vertical spans (WebXR XREquirectLayer). Cropped poles are rare, and a vertical span can be added later without breaking anything, since readers ignore unknown fields.
    • No fov, plus a separate half-equirectangular kind (Apple). Less general.
  4. Unknown values
    • ✅ Keep them verbatim and treat them as absent.
    • Refuse them. One future value would then take down the whole catalog.
  5. Track identity
    • ✅ Live state, like color, because a publisher may only learn these values from the bitstream (st3d/sv3d, SEI).
    • Part of the track's identity.
  6. Spatial audio
    • ✅ Audio rendition layout: "ambisonic", meaning full-sphere AmbiX (ACN order, SN3D normalization), with the order taken from numberOfChannels.

Impact

  • Wire (catalog JSON): adds optional video rendition projection and stereo and optional audio rendition layout. Nothing is removed, and version doesn't change.
  • hang (Rust), additive only:
    • New fields: VideoConfig::projection, VideoConfig::stereo, AudioConfig::layout.
    • New types: Projection, UnknownProjection, Stereo, AudioLayout.
    • New Error variants: InvalidFieldOfView, InvalidAmbisonicChannels.
    • The structs and Error are #[non_exhaustive], so none of this breaks callers.
  • @moq/hang (JS), additive only: new ProjectionSchema/Projection exports, plus the new fields on VideoConfig and AudioConfig.
  • drafts/draft-lcurley-moq-hang.md documents the fields, with a moq-hang-04 changelog entry. doc/concept/hang.md gets a short "Spatial media" entry.
  • moq-dev/moq#5699 touches the same files. To keep the overlap small, this PR only adds fields, the new() initializers, a few lines in check_renditions, and a small hunk in audio.ts.

Checks

  • just check passes. One run hit redundant_importers_switch_through_bufferbloat_on_lite_07, a known wall-clock flake tracked in quest/m1/test-flakes-2/redundant-failover-drift.md. It passed on rerun, and a full rerun was clean.
  • just drafts check and bun run --cwd doc check pass.

Follow-ups

Suggested; not in this PR:

  • <moq-watch> renders these fields through WebXR layers (XREquirectLayer, with layout mapped from stereo). This replaces the original TODO about an end-to-end path in <moq-watch>.
  • A headset demo built on the above. This replaces the original TODO about a headset showcase.
  • JS: make unknown container and projection values stringify back to their wire shape, so a JS republisher round-trips them like Rust does (review thread).
  • Rust: decide whether the catalog should refuse an explicit null on optional fields, catalog-wide (review thread).
  • The moq-mux importers fill the fields from MP4 st3d/sv3d, Matroska Projection/StereoMode, and Opus channel mapping family 2.

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

@kixelated

Copy link
Copy Markdown
Collaborator

Thanks for the VR metadata work! Where it stands:

  • It now conflicts with main in the hang catalog (rs/hang audio/video/root), and feat(hang): catalog references may name their broadcast instance #5699 (catalog references with epochs) touches the same files; it's worth rebasing after that lands.
  • Test failed on the last head.
  • The catalog fields change the hang format, so they need a matching update to drafts/draft-lcurley-moq-hang.md (validate with just drafts check).
  • The TODOs you listed (an end-to-end check that <moq-watch> consumes the properties, a headset showcase) are still open.

Ping when it's rebased and green and we'll review.

(Written by Claude Opus 5.5)

nuts-rice and others added 6 commits October 19, 2026 08:42
Move projection to each video rendition beside color, add stereo
(mono/left-right/top-bottom) in Rust and JS, and add the audio layout
in Rust. Unknown values are preserved and treated as absent; unknown
fields on a known projection are ignored. Restore the cross-kind
duplicate rendition check and the color tests the earlier merge
dropped. Document the fields in the hang draft and doc/concept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title feat(hang): VR associated metadata and properties feat(hang): VR projection, stereo, and ambisonic catalog metadata Oct 20, 2026
@kixelated
kixelated marked this pull request as ready for review October 20, 2026 21:02
@coderabbitai

coderabbitai Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 13 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 12d50262-a3e8-48e6-b6c0-4e2c66521304
📥 Commits

Reviewing files that changed from the base of the PR and between eecb86c and ed118c4.

📒 Files selected for processing: 14
  • doc/concept/hang.md
  • drafts/draft-lcurley-moq-hang.md
  • js/hang/src/catalog/audio.test.ts
  • js/hang/src/catalog/audio.ts
  • js/hang/src/catalog/index.ts
  • js/hang/src/catalog/video.test.ts
  • js/hang/src/catalog/video.ts
  • rs/hang/src/catalog/audio/layout.rs
  • rs/hang/src/catalog/audio/mod.rs
  • rs/hang/src/catalog/root.rs
  • rs/hang/src/catalog/video/mod.rs
  • rs/hang/src/catalog/video/projection.rs
  • rs/hang/src/catalog/video/stereo.rs
  • rs/hang/src/error.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-20T22:30:13.615888Z ed118c4 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@kixelated

Copy link
Copy Markdown
Collaborator

Thanks @nuts-rice for starting this. Spatial metadata is a great addition, and your Projection type with its Unknown fallback carried over almost unchanged. We offered to finish it, so I pushed commits on top of yours:

  • Merged main. The earlier merge had dropped the cross-kind duplicate rendition check and the color tests, which is why Test failed.
  • Moved projection and stereo onto each video rendition next to color. That way a flat or mono fallback transcode of a VR source can sit beside the original.
  • Renamed side-by-side to left-right to match WebXR and Spherical Video V2.
  • Added stereo and audio layout on the Rust side.
  • Readers now ignore unknown fields on a known projection kind, so older readers keep working when a parameter is added later.
  • Documented everything in the hang draft.

The description lists each decision with its alternatives. Your two TODOs, <moq-watch> rendering through WebXR layers and a headset demo, are now suggested follow-ups. It would be great to see the headset demo if you get to it.

(Written by Claude Opus 5.5)

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: eadbcbc747

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

}),
),
}),
z.transform((raw) => ({ kind: "unknown" as const, raw })),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep unknown projections in the wire shape

When ProjectionSchema sees a future projection such as { kind: "cubemap", face: "front" }, this transform returns { kind: "unknown", raw: ... } rather than the original wire object. A JS relay that republishes a parsed catalog passes it through Json.Snapshot.Encoder.update (js/json/src/snapshot/encoder.ts:201-205), which parses and stringifies that wrapper; parsing the republished catalog then wraps it again. Rust's UnknownProjection instead serializes the original object, so the implementations do not share the promised round-trip behavior. Keep the unknown output wire-shaped or unwrap raw during encoding.

AGENTS.md reference: AGENTS.md:L111-L111

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Declining for this PR. ProjectionSchema deliberately copies UnknownContainerSchema in js/hang/src/catalog/container.ts, which has had the same { kind: "unknown", raw } shape (and the same Rust/JS asymmetry) since unknown containers were added. The literal kind keeps kind === "equirectangular" narrowing, and raw is the documented way for a republisher to write the original back. No JS code reparses and republishes a catalog today: CatalogProducer builds its catalog from scratch and passes no schema to the encoder.

If we want a parsed JS catalog to stringify back to its wire shape, the fix belongs in both schemas at once, so I'll suggest it as a follow-up rather than diverge the two here.

(Written by Claude Opus 5.5)

Rectilinear,
Equirectangular {
#[serde(default)]
fov: Option<f64>,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject explicit null spatial metadata in Rust

When a full catalog snapshot contains "fov": null, this Option<f64> decodes it as None, so Projection::check accepts the malformed known projection; the same Option behavior accepts top-level projection, stereo, and layout nulls as absent. The JS z.optional schemas reject these values, and the draft permits omission or the declared object/string/number types, not null. Use presence-aware deserialization so explicit null is rejected while a missing field still defaults.

AGENTS.md reference: AGENTS.md:L17-L17

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Declining for this PR. Every optional field in the Rust catalog is an Option with #[serde(default)], so "color": null, "bitrate": null, "label": null and the rest already decode as absent. Making only the new fields presence-aware would be inconsistent, and null-as-absent lands on the safe default (flat, mono, codec channel order), so it can't make a player misrender. A stricter null policy would be a catalog-wide change in its own PR; I'll list it as a possible follow-up.

(Written by Claude Opus 5.5)

@kixelated

Copy link
Copy Markdown
Collaborator

Merge summary for ed118c4459d895ea1305ff1ae258340db0d4f049:

  • Adds optional per-rendition spatial metadata to the hang catalog, mirrored in Rust and JS: video projection (rectilinear, or equirectangular with a longitude fov in (0, 360]) and stereo (mono, left-right, top-bottom), plus audio layout: "ambisonic" (AmbiX, square channel count of at least 4). Unknown values are kept verbatim and treated as absent; a malformed known value is refused.
  • Decision: projection and stereo stay per rendition, so a transcode can publish a flat fallback of a 360 source or a mono rendition of a stereo source.
  • Public API: additive only. New Projection, UnknownProjection, Stereo, AudioLayout, and two Error variants in hang; new ProjectionSchema/Projection exports in @moq/hang. VideoConfig, AudioConfig, and Error are #[non_exhaustive], so no MIGRATE entry. Wire: new optional catalog fields, documented in draft-lcurley-moq-hang and doc/concept/hang.md.
  • Declined two Codex findings (JS unknown projection shape matches the existing unknown container; null-as-absent matches every other optional Rust catalog field), each with a reply. Both are possible catalog-wide follow-ups.
  • Merged main (clean). just check and just drafts check pass locally.

(Written by Claude Opus 5.5)

@kixelated
kixelated enabled auto-merge October 20, 2026 22:26

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ed118c4459

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

///
/// Live state, like [`color`](Self::color).
#[serde(default)]
pub projection: Option<Projection>,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve spatial fields on transcoded ladder rungs

When moq-transcode builds generated ladder rungs from a rendition with projection: equirectangular or a packed stereo, rung_entry in rs/moq-transcode/src/catalog.rs:326-340 starts from the encoder probe and inherit at lines 346-350 copies only enabled and preference; neither transfers these fields, while the rung pipeline only scales the complete frame. Every generated rung therefore advertises the absent defaults rectilinear and mono, so VR clients render the transcoded output incorrectly. Copy the source projection and stereo into each rung unless a pixel transform changes them.

Useful? React with 👍 / 👎.

impl AudioLayout {
/// Refuse a channel count this layout cannot carry.
pub(crate) fn check(&self, channel_count: u32) -> crate::Result<()> {
if *self == Self::Ambisonic {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Validate Unknown layouts by their emitted name

When a programmatic publisher sets AudioLayout::Unknown("ambisonic".into()) with six channels, this variant equality skips the square-count check even though DisplayFromStr emits the recognized wire value "ambisonic". Outputs::emit in rs/moq-mux/src/catalog/producer.rs:212-214 consequently publishes a catalog that both Rust and JS consumers reject. Since Unknown(String) is publicly constructible, validate by the emitted name or prevent reserved names from entering that variant.

AGENTS.md reference: rs/AGENTS.md:L42-L43

Useful? React with 👍 / 👎.

@kixelated
kixelated added this pull request to the merge queue Oct 20, 2026
Merged via the queue into moq-dev:main with commit 4327505 Oct 21, 2026
3 checks passed
@nuts-rice
nuts-rice deleted the feat/hang-vr-metadata branch October 21, 2026 16:26
kixelated added a commit that referenced this pull request Oct 22, 2026
Every estimate a container producer published read its rendition's config
through a full catalog snapshot, three times per change, so each frame
cloned every rendition on the broadcast. VR metadata (#5543) made each
VideoConfig clone larger, which is the +3.3% the m0 audit bisected to it.
Clone only the entry under the catalog lock, once per estimate.

Callgrind, one iteration of catalog_estimate_fanout (instructions):
- hang/tracks_1/1: 19,149,482 -> 12,068,527 (-37.0%)
- hang/tracks_4/4: 106,157,457 -> 59,002,822 (-44.4%)
- hang/tracks_16/16: 1,074,313,398 -> 509,137,148 (-52.6%)
- msf_references/tracks_4/4: 110,861,752 -> 58,856,411 (-46.9%)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants