Support more audio and video codecs

[MathieuDebit]

What version of WebTorrent Desktop? (See the 'About WebTorrent' menu)
Version 0.20.0 (0.99.3)

What operating system and version?
Windows, MacOS and Linux

Support more codecs

I'd like to discuss about ways to support more audio and video codecs in Webtorrent Desktop. Considering the choosen technology (Electron/Javascript), there are different solutions available (FFMPEG.js, WebAssembly, plugins, etc), but I suppose the more relevant comes from Butter Project.

Butter Desktop currently supports native decoding for a lot of audio and video codecs (see supporting codecs). I tried to dive into the code, I'm not sure about it but they seem to use video.js with a set of addons and custom implementations.
In the end, Butter Desktop is an app built upon similar web technologies, but supports all the audio and video codecs needed.

What do you think about it? Do you think we could be helped by the current Butter Desktop codebase to support more codecs in Webtorrent Desktop?

Thanks a lot 🙏

[feross]

@MathieuDebit Any ideas how they support the AC3 audio codec? Are they running ffmpeg manually? Or did they bundle the codec into the chromium content module?

[MathieuDebit]

@feross I'm not really sure as the codebase goes beyond my skills, but Butter is still using NW.js, and it seems that there is a way to enable prioritary codecs by compiling sources with FFMPEG:

• nwBuilder withFFmpeg: true: https://github.com/butterproject/butter-desktop/blob/caed8c45e90f471587225d5e3744bc006e1537e3/gulpfile.js#L115-L125
• "Nearly impossible to understand how to compile ffmpeg": Nearly impossible to understand how to compile ffmpeg nwjs/nw.js#4595
• "enabling for ac3 and hevc codecs support": enabling for ac3 and hevc codecs support nwjs/nw.js#5653
• "Enable Proprietary Codecs" on NW.js: http://docs.nwjs.io/en/latest/For%20Developers/Enable%20Proprietary%20Codecs/

[MathieuDebit]

@feross as you said here #350 (comment), I guess the best solution would be to build Electron with a custom FFmpeg, did you already explore this approach?

• Here is an idea of how to implement it: Building with an alternative libffmpeg electron/libchromiumcontent#198 (comment)
• It seems that there is no problems with seeking: Building with an alternative libffmpeg electron/libchromiumcontent#198 (comment)
• We should also think about legal issues by adding proprietary codecs: Provide ffmpeg.so file that plays all common videos electron/electron#9534 (comment)

Sources:

• https://discuss.atom.io/t/information-about-bundled-ffmpeg/28456
• Building with an alternative libffmpeg electron/libchromiumcontent#198
• http://pietropassarelli.com/ffmpeg-electron.html
• Provide ffmpeg.so file that plays all common videos electron/electron#9534

[feross]

did you already explore this approach?

Yep, we did at one point in time but abandoned it because it complicated the build process too much and the legal situation was unclear. I think things have gotten better on both fronts, so happy to explore this in future versions of WD.

If anyone else has the interest/skill to tackle this before then, go for it!

[MathieuDebit]

So the idea is to replace the ffmpeg library in Electron with our own one.

Thankfully projects like ffbinaries or ffmpeg-static allow us to get the latest binaries for all platforms.

But I don't know how to tell Electron to use these binaries instead... 🚶

Also it could change the build process, will we need a CI or something like electron-builder?

[MathieuDebit]

Sorry if all of this seems obvious for some of you, but as I'm still learning in that field, I prefer write everything down to be sure the most noob of us will be able to follow 😛

So here is two approaches I've found to support more codecs in Electron with FFmpeg:

• build libchromiumcontent with a custom libffmpeg: this solution seems to be the best, but is a power-user feature and means that the WD build process will be way more complicated (here, simply use FFmpeg binaries in Node via ffbinaries or ffmpeg-static (example) is not enough, since we don't only want to access FFmpeg methods, but use it in the <video> tag).

• Convert on-the-fly videos through a local http server like in this app called StreamStudio: this solution doesn't change the build process, and is apparently not that CPU consuming, but will surely complicate the codebase.

I'm digging into the first solution, which is the cleanest, but surely the hardest path 🚣‍♀️

[feross]

@MathieuDebit Thanks for sharing your progress. At one point, I believe the Electron team was producing builds of libchromiumcontent which included a much larger list of ffmpeg codecs enabled. You might want to see if there's something already compiled that you can use.

[MathieuDebit]

Hi @paxter, unfortunately I wasn't able to build Electron with a custom FFmpeg version, because custom builds of Electron and libchromiumcontent are not easy (at least for me 🤷‍♂️) and Electron team does not share their build process...

[dsernst]

Missing codec support can be a bit annoying because using an external player is a little shaky if the file is still downloading.

Right now the error message looks like:

1. As a user, it would be helpful if the error showed the name of the unsupported codec, so I have more information if it's the same type I'm running across often, or more of a long-tail problem.

2. Would it be helpful to start tracking unsuccessful play attempts across different codecs in telemetry logs, to prioritize formats? Looks like https://webtorrent.io/desktop/telemetry is behind a password, though.

[tio-trom]

Shouldn't the player use external codecs installed on the OS? I get this error all the time. Also, shouldn't Webtorrent try to play the video with the default video player from that OS instead of always suggesting to use VLC? Many people don't use VLC and are then faced with installing a new video player.

Would be simpler to have the video directly open in the default video player if a codec error is encountered.

[wellington1993]

I have the same problem here:

https://gist.github.com/wellington1993/fbc4af44951da0aeb499a02cac8a9f14

Audio #1
ID : 2
Format : AC-3
Format/Info : Audio Coding 3
Commercial name : Dolby Digital
Codec ID : A_AC3

[wellington1993]

I fount that project: https://github.com/DanielHuisman/ac3.js/
"JavaScript AC-3 (ATSC A/52) decoder"

The same info found in "Totem" player: AC-3 (ATSC A/52)

maybe it will help, since we are talking about two projects with the same technology.

[WINOFFRG]

Hello! Do we have any updates on this? Can we check this out https://github.com/Muril-o/electron-chromium-codecs
This one bascially patches the ffmpeg bundled with electron and electron as well.

I tried this but faced some issues while building it while also referring to the official build guide.

Kindly share if anyone has updates on this? Also if HEVC has some legal concerns can't we just add AC3 since its patent is expired?

[github-actions]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

[mifi]

FYI Electron now supports HEVC playback when hardware decoder is available. see discussion mifi/lossless-cut#88

[github-actions]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?