[CODEC] Hardware accelerated video decoding/encoding

[SebastianTheFish]

do it

[itsmattkc]

Presently, I don't think I would use the existing FFmpeg library for this because there doesn't seem to be a (documented) way to decode an H.264 frame and draw an OpenGL texture with it. The official NVIDIA NVDEC documentation states very clearly how to do this, so it seems like using that directly would be a better solution.

[alcomposer]

Currently is there any Hardware decoding / encoding (mainly decoding) for ProRes or other high quality editing formats? (not in Olive, but available as a library etc.)

I am aware that Apple has an private API for accelerating ProRes, which obviously isn't available, but I don't know of any for DNxHD or similar. Hardware decoding for intermediate media would be very useful when Olive gets the ability to render material in the timeline, speeding up the process and also quickly zooming over media.

[itsmattkc]

As far as I know there isn't, NVENC/NVDEC seem to be solely for H.264 and HEVC. Perhaps it would be possible for someone to write a decoder for OpenCL, but I don't think it exists at the moment.

[alcomposer]

For an optimised internal media codec, possibly Cineform could solve this issue. It is highly optimised, (much more than ProRes) and it’s also MIT licensed.

Like other NLE’s there could be an option to optimise media in bins, which would initiate a conversion of selected media into Cineform for editing.

Although I am not aware if Cineform is GPU accelerated, internal usage could provide a substantial speed up, especially with modern CPU’s.

[eszlari]

FFmpeg's ProRes decoder is multi-threaded and afaik, the format is not that complex.

[elsandosgrande]

It would be nice if possible to include AMD hardware support (theoretically even dual-GPU things down the line, as my laptop's APU paradoxically supports many more codecs in hardware than the dedicated GPU, so that's just lovely).

[capezotte]

Presently, I don't think I would use the existing FFmpeg library for this because there doesn't seem to be a (documented) way to decode an H.264 frame and draw an OpenGL texture with it.

h264-vaapi provides hardware acceleration for thr first half of the process, though I'm not sure about the "OpenGL texture" thing.

[itsmattkc]

I'm aware FFmpeg has hardware accelerated encoders, but the important part is directly interfacing with OpenGL. Otherwise the (most likely) workflow would be transferring a packet to the GPU for decoding, transferring the decoded frame back to the CPU, and the CPU transferring the frame back to the GPU as an OpenGL texture. The overhead of transferring data back and forth negates any performance improvement.

At one point Intel QuickSync was implemented, but showed little to no improvement because of a workflow similar to this. A lot of hardware accel libs can interface with OpenGL (notably NVENC/NVDEC), but it doesn't seem FFmpeg exposes this functionality meaning the libs will likely have to be implemented manually.

[elsandosgrande]

A lot of hardware accel libs can interface with OpenGL (notably NVENC/NVDEC)

Umm, would that also include VAAPI/VDPAU? I only really know of their use in VLC, and it doesn't seem to be bottlenecked anywhere, so I would assume as much.
If that IS the case, how realistic would it be to implement within one to two years (if that works like that and is implemented, it would open it up to those not using Nvidia's silicon, so here's hoping)?

[itsmattkc]

@elsandosgrande From what I can tell, yes VAAPI has a system of encoding/decoding H.264 frames to OpenGL textures. The only equivalent I could find with QuickSync was interfacing with DirectX textures which won't work for Olive (but perhaps there's another way to do those). All I'm saying is it seems all of these will have to be implemented outside of FFmpeg which is unfortunate (unless somewhere more familiar with FFmpeg knows more about this).

[elsandosgrande]

From what I can tell, yes VAAPI has a system of encoding/decoding H.264 frames to OpenGL textures. All I'm saying is it seems all of these will have to be implemented outside of FFmpeg which is unfortunate (unless somewhere more familiar with FFmpeg knows more about this).

Well then...
One, yes, there is hope!
Two, guess I better get acquainted with FFmpeg's codebase then.

[alcomposer]

I believe that Nvidia can now decode/encode h264 & h265 directly on the GPU.
In Davinci Resolve, Blackmagic enabled this for Nvidia cards (obviously):
https://forum.blackmagicdesign.com/viewtopic.php?f=21&t=82235

[elsandosgrande]

I believe that Nvidia can now decode/encode h264 & h265 directly on the GPU.

Any word on AMD cards?

[itsmattkc]

NVENC/NVDEC has been a thing for quite some time, but it looks like Resolve has just implemented it. It'd be very cool to have this, but as I said before I'm not sure I can get it (and libs like it) working with the current FFmpeg workflow.

Any word on AMD cards?

I think VAAPI/VDPAU is still our best bet. The only OpenCL-based H.264 is proprietary and expensive to license.

[elsandosgrande]

@itsmattkc Wait, is the only one referring to Nvidia or AMD?

[itsmattkc]

@elsandosgrande I was mostly talking about AMD, but it does seem VDPAU can provide some acceleration for NVIDIA cards too.

[alcomposer]

I'm not sure I can get it (and libs like it) working with the current FFmpeg workflow.

Moving towards an internal design that disconnects FFmpeg from input/output is probably a good idea then. (I'm not saying to get rid of FFmpeg) Being able to use FFmpeg to get footage into Olive, then have Olive do what it needs to the footage, (editing/effects) then exporting using independent libraries / backends. I think that implementing a caching system would go a long way to detaching FFmpeg, per the discussion in #237

[itsmattkc]

I've actually seen an example of being able to transfer OpenGL textures to CUDA memory to be encoded by FFmpeg. At least as far as NVENC goes, this may be a lot easier than initially expected. Maybe this is the same for VAAPI/VDPAU.

[alcomposer]

It would be a good idea, Nvidia is pushing NVENC technology. However this would also leave Intel & AMD users out in the dark?

[itsmattkc]

That's hopefully what VAAPI/VDPAU will help address if the same thing can be done for that API, but yes NVENC is naturally NVIDIA exclusive.

[unfa]

+1 for supporting Cineform

[alcomposer]

I've cooled down on the idea of supporting Cineform, however what other decoders apart from FFmpeg (libav) are there on the Linux front? (or more specifically OSS front)

There are no other decoders for ProRes, DNxHD etc. For H264 / 5 yes, probably NVENC but that is basically all.

[elsandosgrande]

@alcomposer coughVAAPIcough

[Simran-B]

I wonder if OBS supports QuickSync only on Windows, where it uses a DirectX renderer (for encoding though). There is support for AMD cards through https://github.com/obsproject/obs-amd-encoder - probably limited to h.264?

[ric96]

OBS supports VAAPI now https://www.phoronix.com/scan.php?page=news_item&px=OBS-Studio-VA-API-H264

[xenogenesi]

h264_nvenc even if only for export it would be nice, I use it with ffmpeg and it makes the difference.

[sobotka]

Many GPUs screw up the encoding if done using their API paths. It wasn’t uncommon for example, for video sites to have the wrong decoding when flipping to full screen GPU. Clueless monkeys at the GPU driver level shouldn’t surprise anyone.

[xenogenesi]

I tried a raw patch over the 0.1.x branch, I added a dummy codec in export, video only, which uses popen () to launch ffmpeg with rawvideo input and h264_nvenc output.

The encoding of a half-hour video has gone from half an hour to less than 15 minutes (but without audio, and probably I should check other quality factors), but the most interesting difference for me is that the CPU load remains almost free and does not heat up at 90° for half an hour (the GPU reaches at most only 50°/60°).

I believe that for export, launching a process could be interesting, it's simple to make and there is no need to link libraries

[mcdonc]

Thanks so much for Olive. I've used it to produce something like 40 videos, and it sucks so much less to do so than using anything else that I'm very grateful for it.

H.264 (and, less-so H.265) hardware-accelerated encoding on export is the only feature I miss from other things I've used. I'm unfamiliar with the new architecture (I've been using 0.1) but, if it makes any sense at all (which it may not), it might not be a terrible idea to expose a "renderer " (defaulting to "libx264", I suppose) which is "ffmpeg" such that it just pipes the raw video and audio out, providing a slow, solid cpu-bound ffmpeg rendering using conservative flags, and leaves it to the user to configure the right extra flags to get accelerated encoding working with whatever wild hardware setup they have. I'm sure this isn't an original idea, but doing so would let us take advantage of our hardware while some VAAPI/VDPAU solution is worked out in good time without necessarily tying Olive forever to ffmpeg.

[bertin0]

I think VA-API would be the most universal solution, as there is now (for a while actually) a VA-API driver for Nvidia cards that uses NVENC/NVDEC under the hood.

[xenogenesi]

I think VA-API would be the most universal solution, as there is now (for a while actually) a VA-API driver for Nvidia cards that uses NVENC/NVDEC under the hood.

I was using ffmpeg built with cuda/nvenc support for the encoding, for the branch 0.1 using popen was such an easy patch (I had some define to use Qt classes instead of popen for cross compatibility but never tested on Windows/Mac).

[jorsn]

Any updates? For me, this is a real blocker. I can't sensibly use this software if I can't do hardware encoding. Playing with the ffmpeg options once is better than waiting always.

[sobotka]

Then use other software.

Olive was largely built by one person, who could not really keep at it.

Hardware encoding also degrades the output.

https://www.patreon.com/posts/olives-future-79926246