Repository navigation
Feature request: Add side_data attribute to Stream #1150
Description
Overview
Sometimes a stream encodes side data in the entire stream (in side_data). Some of those side data are quite essential to read the video correctly.
For instance displaymatrix sometimes encode whether the video is rotated or not.
From the AV documentation it seems like we could know easily if there is side data with the attribute nb_side_data.
I would like to know if this is an interesting feature for this library ? I started a fork there with, for now, a simple skeleton of how it would look like
Desired Behavior
When reading a video (or any other stream) you would get the side data as a dictionary:
# This would give you the stream metadata
container.stream.video[0].side_data
Activity
hyenal commentedon Aug 9, 2023
I made this PR on my own fork. It only fetch the DISPLAYMATRIX but would be happy to implement more and submit a PR to this repo if anyone is interested
lgeiger commentedon Nov 21, 2023
@hyenal I'd be really interested in having this functionality upstreamed to
avas it would fix #1045 is a blocker for me currently and also many people seem to run into.@jlaine or @WyattBlue are probably in a better position to judge how this should be implemented best to make it consistent with the way frame side data is handled. I'm happy to help out and write some test cases if that would be helpful.
hyenal commentedon Nov 21, 2023
Thanks @lgeiger, so happy to see this getting momentum!
It looks like it's been merged in @WyattBlue repository and will be available in PyAv soon?
I am also happy to write some tests if this helps :)
lgeiger commentedon Nov 21, 2023
Yes, it's already available on pypi though this causes dependency conflicts for me as I am relying on a different package that requires
avdirectly which is why it would be great to get upstreamed at some point.hyenal commentedon Nov 22, 2023
I see thanks for the explanation!
Let's wait for an answer from maintainers then, I am hopeful this could be merged 🙏
WyattBlue commentedon Nov 25, 2023
Sure, @hyenal Make a new PR for this repo.
hyenal commentedon Nov 27, 2023
@lgeiger do you have some tests in mind ? As far as I understood we rely on FATE to get video samples, I hope those have a sample with a non zero display matrix so that we could write a test easily.
Otherwise the remaining work I had in mind was to make sure stream.sidedata works and polish the code (removing the two TODOs)
I intend to do this next week so if you it's a blocker for you I d be happy to invite you as a contributor to my PR :)
lgeiger commentedon Nov 27, 2023
Not really. I probably could share some example videos but I guess it would be better to get them from FATE.
That's great, thanks for working on it. It's not a blocker for me anymore. For reference, we ended up using pymediainfo to read the rotation information from the video files and handle them accordingly via Numpy which works well for our use case but it would still be great if this would be natively supported in av.
WyattBlue commentedon Oct 10, 2024
We're planning to remove Stream.side_data (#1570) in PyAV 14 since FFmpeg doesn't even document
AVStream.side_dataorAVStream.nb_side_dataanymore.I think there will be some alternative API, but we're waiting for FFmpeg 8 to be released.
lgeiger commentedon Nov 12, 2024
@WyattBlue thanks for the heads up. What's the recommended way of checking whether the video content needs to be rotated with PyAV?
As far as I can tell without
side_datawe won't know whether the frame needs to be rotated (e.g. when we need to do further processing of the numpy frame).Of course this can be checked via
ffprobe -loglevel error -select_streams v:0 -show_entries stream_tags=rotate -of default=nw=1:nk=1 video.mp4or using something like pymediainfo but that means adding another requirement which tends to complicate things.As far as I can tell correct handling of rotation is the reason why some upstream packages still rely on av < 10 (see #1045).