[RFC] Hardware-accelerated video decoding

[bryandeng]

🚀 Feature

Hardware-accelerated video decoding

Motivation

Now that torchscriptable transforms natively supporting GPU have landed, hardware-accelerated video decoding may further help relieve the IO bottleneck commonly seen in large-scale video deep learning tasks.

Pitch

This functionality is likely to be built upon FFmpeg's hardware acceleration APIs, since FFmpeg is already in use and it's easier to support multiple hardware platforms and platform APIs in this way.

Alternatives

Decord and NVIDIA VPF are both PyTorch-friendly video IO libraries which support (NVIDIA only) hardware-accelerated video decoding to some extent.
NVIDIA VPF is built upon NVIDIA Video Codec SDK directly without FFmpeg.

Additional context

https://trac.ffmpeg.org/wiki/HWAccelIntro
https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new

[fmassa]

Hi,

Thanks for opening the issue!

I agree that video decoding on the GPU would be a nice functionality to have, although it's not in the near-term plan for now (maybe in 6 months?).

About seeking on videos, we are preparing a revamp of the video reading abstractions which will be more generic and allow for more flexibility while performing video decoding. cc @bjuncek

For video transformations on the GPU, note that we are making all transforms in torchvision to be torchscriptable and work with Tensors, so that they will natively support GPU if needed.

[bryandeng]

Updated description after the release of torchscriptable transforms.

[bryandeng]

We may first

• discuss which hardware platforms to support
• find out the constraints on input video formats and output pixel formats, etc.

[JuanFMontesinos]

Hi,
I wrote this small library which wraps ffmeg and does exactly that.
https://github.com/JuanFMontesinos/PyNVIdeoReader
For those who need it, it can work in the meanwhile.

The only drawback against NVIDIA VPF is numpy arrays are allocated at CPU in the end. However it allows to use all the ffmpeg tools. thus it's way more flexible.

I think something like this can be adapted to the current video reader that torchvision have.

[dwrodri]

Pinging this thread to check for interest. Nvidia has been working on writing their own bridge between their GPU's encoders/decoders and popular machine learning libraries (link here), but it's still in beta. Is there any interest in making this a feature in PyTorch?

I could see something like this introducing some unintuitive edge cases, since an iterator that returns CUDA Tensors may not play well with DataLoaders which leverage multiprocessing. However, the goal of this feature addition would be to offer developers the option to leverage the Nvidia GPU decoders without installing another dependency.

However, if the integration can be done from FFmpeg's C++ API, then it could probably be easily expanded to include Intel's and AMD decoder APIs as well, assuming they're all available form the same API.

[JuanFMontesinos]

@dwrodri
In fact Nvidia offers a c++ API (video codek SDK) which implements this GPU decoding in case anyone wants to create a customized library.
In my case I just found calling FFMPEG was simpler for the problem I wanted to solve.

From my point of view it would be a really interesting tool. Working with videos and cpu is horrible and I just found maaaany issues. DALI works like a charm but it's not very flexible.

[dwrodri]

Most Python libraries I've seen that interface with the decoder onboard an Nvidia GPU pawn it off to FFmpeg's CLI using subprocess. There's nothing inherently wrong with this, and is definitely the most "quick and dirty" way to get it done. Unfortunately, I don't think this approach would suffice for PyTorch most users would want to access the decoded frames as a CUDA Tensor as they exit the decoder.

libAV, the library form of FFmpeg, offers a wrapper around the propprietary hardware accelerators they support. If I can find the time, I'd like to see if I can put together a PR which would allow someone to pass a torch.device to the torchvision.io.VideoReader.__init__() call, and then set up the backend accordingly using libAV. That way, you could use FFmpeg's bindings for AMD's AMF/VCE, which is especially relevant because of PyTorch's recently-added Rocm support. Of course, all of this easier said than done.

DALI doesn't support variable-frame-rate video, which is unfortunate for me because I work with video containing missing frames quite often.

[JuanFMontesinos]

I agree that it would be nice but it's not straight forward. The support varies a lot across different GPU models.
Besides, the video format is really specific: mp4 yuv 4:2:0 for decoding and mp4 420,422 or 444 for encoding (and depending on the gpu model)
More info here

In short, if the user want to use GPU decoding it's really likely will have to prepare the videos beforehand. I find much simpler to pass a flag and let the user to fall in the codec exception if the format is wrong.

[JuanFMontesinos]

Let me just add that DALI's fixed-frame-rate support is not that bad.
It allows to disect a video precisely ensuring proper seeking and audio-visual synchronization.
Different libraries address missing frames in different ways so that you can find they return different amount of frames.