Skip to content

feat(mux): fill the catalog's VR fields on import - #7308

Merged
moq-bot[bot] merged 12 commits into
mainfrom
quest/m2/mux-vr-fields
Oct 31, 2026
Merged

moq-bot[bot] merged 12 commits into
mainfrom
quest/m2/mux-vr-fields

Conversation

@kixelated

@kixelated kixelated commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Finishes quest/m2/mux-vr-fields.md: the moq-mux importers fill the hang catalog's spatial fields from moq-dev/moq#5543.

  • MP4 (fMP4): every visual sample entry (avc1/avc3/hvc1/hev1/vp08/vp09/av01) reads Spherical Video V2 st3d/sv3d through mp4-atom 0.17's spatial field (fmp4::spatial), like colr and pasp.
  • Matroska: StereoMode and Projection (ProjectionType, ProjectionPrivate, ProjectionPose*) fill the same fields (matroska_spatial).
  • fov: shared codec::spatial::equirectangular derives the catalog fov from the equi bounds (0.32 fixed point): VR180 bounds of 0.25 each give 180, full coverage leaves it absent (360).
  • Opus: channel mapping family 2 imports as layout: "ambisonic" (From<opus::Config> for AudioConfig), so raw Opus, Matroska, FLV, and TS OpusHeads all get it. opus::Mapping::new refuses family 2 at zeroth order, with RFC 8486's non-diegetic stereo pair, or with a count RFC 8486 forbids (UnsupportedChannelCount).
  • Refused (fmp4::Error::UnsupportedSpatial, mkv::Error::UnsupportedSpatial): cubemap, mesh (including Google's mesh VR180), unknown projections, nonzero pose, asymmetric or vertically cropped bounds, bounds leaving no picture, and stereo packings other than mono, left-right, and top-bottom with the left eye first (st3d right-left/custom; Matroska right-eye-first, checkerboard, interleaved, anaglyph, laced). Also refused in MP4: a repeated or undecodable st3d/sv3d/proj/projection box, and an sv3d without proj.
  • mp4-atom 0.17.0: this branch merges moq-dev/moq#7043 (counted trun samples), swaps its git pin for the now-released crate (reverting its deny.toml allow-git), and adapts to the other 0.17 breaks: TrunEntry::cts is i64, EsDescriptor has new optional fields, and Dref keeps typed entries. Codex's review also found that feat(mux)!: adopt mp4-atom's counted trun samples #7043's segment keyframe scan reset its mdat budget per traf, so it now spans the whole moof.
  • Quests: deletes this quest and its links, rewrites fMP4 ambisonic layout (now unblocked except for CMAF surround Opus) and CMAF surround Opus to point at the release. It also carries feat(mux)!: adopt mp4-atom's counted trun samples #7043's deletion of quest/m1/cut-audit/fmp4-sample-resolver.md.

Public API: UnsupportedSpatial variants on the non-exhaustive fmp4::Error and mkv::Error, plus the breaking mp4_atom 0.17 re-export (CHANGELOG updated). MoQ wire: none.

Decisions

Decided by the maintainer on 2026-10-30.

Choices made without asking

  • Full 360° coverage leaves fov absent (the catalog's "absent means 360") rather than writing 360. st3d mono and Matroska StereoMode 0 leave stereo absent, and a rectangular Matroska ProjectionType 0 leaves projection absent.
  • A Matroska equirectangular Projection without ProjectionPrivate covers the whole sphere, as FFmpeg reads it. A private payload that isn't a version 0 equi body (20 bytes) is refused.
  • Nonzero Matroska pose is refused for every ProjectionType, including rectangular, since it would rotate the picture. A NaN pose counts as nonzero.
  • The family 2 check lives in opus::Mapping::new, so it applies wherever an OpusHead is parsed (including the MSF consumer and exporters), not just the importers. That keeps From<opus::Config> infallible.
  • Ignored, as the Plan scopes out: Apple vexu/hfov, Matroska's deprecated OldStereoMode, and codec-level stereo SEI.

Testing

  • just check origin/main: passes on 8aad742a58 (flake skipped: the local Nix builder can't create user namespaces). After the review fix: cargo clippy -p moq-mux --all-targets --all-features -D warnings and cargo test -p moq-mux --all-features (1877 passed) pass.
  • New tests:
    • fMP4 spherical_boxes_fill_projection_and_stereo covers 360 mono, VR180 left-right with equi bounds, and flat top-bottom, across every sample entry.
    • fMP4 spherical_boxes_the_catalog_cannot_describe_are_refused covers cubemap, mesh VR180, asymmetric bounds, vertical crop, yaw/roll pose, right-left, custom, a repeated st3d, and sv3d without proj.
    • Matroska projection_and_stereo_mode_fill_the_catalog and ..._are_refused.
    • Matroska opus_family_two_is_ambisonic (first order accepted; 1, 6, and 5 channels refused) and raw Opus family_two_is_ambisonic.
    • Unit tests for codec::spatial and Mapping::new.
    • fieldless_runs_do_not_stall_the_segment_scan gains a 1000-traf case.

Suggested follow-ups

  • Opus channel mapping family 3 (RFC 8486 projection ambisonics) carries a demixing matrix, not a channel table. Mapping::parse still reads it as a table, so it imports as plain channels or fails as an invalid table. It should be refused, or carried as ambisonic.
  • Exporters writing projection/stereo/layout back into sv3d/st3d, Matroska Projection, and family 2 heads (out of scope per the Plan).

Review

  • Reviewer: Codex (quest review), gpt-6.1-sol, medium effort.
  • Round 1 on 8aad742a58: one finding. [P1] the segment keyframe scan's budget reset per traf, allowing CPU amplification. Fixed in 594fa40693.
  • Round 2 on 594fa406934cdfd025954e08e6e54abca1bbd25f: no findings.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

kixelated and others added 11 commits October 28, 2026 00:01
Pin moq-mux to kixelated/mp4-atom#254, which stores a field-less trun
run as a count, so a moof of many such runs decodes without one entry
per sample. Migrate the sample resolver, the segment-start scan, and
the per-track fragment rewrite to `Trun.samples`, and test that a moof
of 1000 u32::MAX-sample runs decodes and resolves only up to the mdat.

Swap the git pin for mp4-atom 0.17.0 once it is published.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An inherited trun run can claim u32::MAX samples in four bytes, so
walking every sample for a sync flag before the resolver bounds them
would trade the old per-sample allocation for a multi-billion step loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mp4-atom now decodes a nonzero dOps family instead of refusing it, so the
fMP4 importer refuses the table itself until surround Opus lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Quest-Claim: 7f161c8f3d778853
Swaps the git pin for the published crate and adapts to the changes
released since: trun composition offsets widen to i64, the ES descriptor
gains its optional fields, and dref keeps typed entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MP4 import reads Spherical Video V2 st3d/sv3d and Matroska import reads
StereoMode/Projection into the rendition's projection and stereo, deriving
fov from symmetric equirectangular bounds. Opus channel mapping family 2
imports as layout "ambisonic". What the catalog cannot describe (cubemap,
mesh, pose, cropped or uncentred bounds, other stereo packings, zeroth
order or non-diegetic stereo ambisonics) is refused.

Finishes quest/m2/mux-vr-fields.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The mdat budget reset per traf, so a moof repeating a forged field-less
run across many trafs multiplied the walk. One budget now spans them all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title moq-mux importers fill the VR catalog fields feat(mux): fill the catalog's VR fields on import Oct 30, 2026
@kixelated
kixelated marked this pull request as ready for review October 30, 2026 08:32
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-30T08:34:22.253812Z 594fa40 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 57 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 234027f0-14e8-465a-8c06-bbd73e7e6895
Commits

Reviewing files that changed from the base of the PR and between d812207 and 9c5c225.

Files ignored due to path filters: 1
  • Cargo.lock is excluded by !**/*.lock
Files selected for processing: 26
  • quest/m1/cut-audit/README.md
  • quest/m1/cut-audit/fmp4-sample-resolver.md
  • quest/m2/README.md
  • quest/m2/cmaf-opus-surround.md
  • quest/m2/mux-vr-ambisonic-fmp4.md
  • quest/m2/mux-vr-fields.md
  • quest/m3/watch-webxr.md
  • rs/moq-hls/src/export/mod.rs
  • rs/moq-mux/CHANGELOG.md
  • rs/moq-mux/Cargo.toml
  • rs/moq-mux/benches/containers.rs
  • rs/moq-mux/src/codec/mod.rs
  • rs/moq-mux/src/codec/opus/import.rs
  • rs/moq-mux/src/codec/opus/mapping.rs
  • rs/moq-mux/src/codec/opus/mod.rs
  • rs/moq-mux/src/codec/spatial.rs
  • rs/moq-mux/src/container/fmp4/edit.rs
  • rs/moq-mux/src/container/fmp4/export_test.rs
  • rs/moq-mux/src/container/fmp4/import.rs
  • rs/moq-mux/src/container/fmp4/import_test.rs
  • rs/moq-mux/src/container/fmp4/mod.rs
  • rs/moq-mux/src/container/fmp4/sample.rs
  • rs/moq-mux/src/container/fmp4/text.rs
  • rs/moq-mux/src/container/mkv/import.rs
  • rs/moq-mux/src/container/mkv/import_test.rs
  • rs/moq-mux/src/container/mkv/mod.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: 594fa406934cdfd025954e08e6e54abca1bbd25f.

No actionable findings from this review. MP4 and Matroska spatial metadata map consistently to the catalog's representable projection/stereo subset, including symmetric longitude bounds and explicit refusal of unsupported pose, crops and packing. Opus family-2 channel validation matches the catalog's ambisonic layout. The earlier local segment-scan amplification finding is fixed: the mdat-derived budget now spans the entire moof rather than resetting per traf, and the sample resolver stops on invalid or overlapping ranges.

Direction: sharing the equirectangular conversion and rejecting unrepresentable metadata is preferable to silently labeling spatial media as flat/plain channels. Keeping fMP4 Opus mappings explicitly refused until the separate mapping-table work lands avoids losing decoder information. The disclosed family-3 parsing limitation is pre-existing and remains outside this change.

Verification limits: static GitHub review, including an independent fMP4 resolver/budget pass. I did not execute tests, media playback, the gate or cross-language interop, nor audit mp4-atom's external implementation; reported passing checks are not independently verified.

(Written by OpenAI)

@kixelated

Copy link
Copy Markdown
Collaborator Author

Landing 594fa406934cdfd025954e08e6e54abca1bbd25f through quest land.

  • CI: 12/12 required checks passed on the head.
  • Reviews: the local Codex review's round 2 on this head had no findings. The online OpenAI review on this head also had no actionable findings.
  • quest check --base origin/main: passes (668 documents). The head merges cleanly with the current main.
  • Decisions: the maintainer's 2026-10-30 picks in the body still apply: this PR supersedes moq-dev/moq#7043, and moq-dev/moq#7336 lands after it.

(Written by Claude Opus 5.5)

@kixelated
kixelated enabled auto-merge (squash) October 31, 2026 04:59
@moq-bot
moq-bot Bot merged commit a116902 into main Oct 31, 2026
7 checks passed
@moq-bot
moq-bot Bot deleted the quest/m2/mux-vr-fields branch October 31, 2026 08:37
kixelated added a commit that referenced this pull request Oct 31, 2026
#7308 moved moq-mux to mp4-atom 0.17.0, so the bump quest no
longer carries 0.17's breaking changes; it waits only on the 0.17.x
release with the sgpd bound.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated added a commit that referenced this pull request Nov 15, 2026
…y fills

#7308 fills projection, stereo, and the ambisonic layout from MP4 st3d/sv3d,
Matroska, and Opus family 2, so the CLI flags and moq_mux::catalog::Spatial
duplicate it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant