Next steps for ffmpeg.wasm

[jeromewu]

Hi,

First of all, thanks for people who use and support ffmpeg.wasm for the past two years. 😄 Without you, it is impossible for ffmpeg.wasm to reach 8.4k+ stars and be used by 1.9k+ projects / packages! And now we also have 32 contributors support this open source project and that's why we have 30 releases already. 🎉

I have to admit I didn't actively maintain this ffmpeg.wasm for the past 1 year, but thanks to this one year I also had time to rethink what is the next for ffmpeg.wasm and I would like to share it here and it would be great if you can provide any feedbacks / suggestions.

There are 4 aspects I would like to share: community, developer experience (DX), performance and features.

Community

As an open source project, it relies on the support from the community to grow. But most of the time the community won't grow itself, it requires mechanisms to achieve. That's why I would like to add / enhance following mecahnisms here:

• ffmpeg.wasm discord channel
• Document roadmap with Github Projects

Developer Experience

As one of the main contributor of this project, I really suffered from the not-so-good DX when developing ffmpeg.wasm. Mainly there are 5 issues I really want to do to make it easier for contributors:

• Adopt monorepo using lerna to keep source code in one place
• Speed up core build time (it takes around 1 hour to build a new ffmpeg.wasm-core) with cache and parameters tuning
• Adopt typescript to achieve one module system and provide types
• CI for building and testing assets.
• Documentation for how to build and contribute to this project

Performance

If you check Issues here, many issues actually relate to performance issue of ffmpeg.wasm. That is one issue I found it extremely critical and hard to resolve. But if we don't do anything about it, ffmpeg.wasm will never reach a state of usable.

The primary attempt was to use multi-threading to speed up, it works but also suffers from comapatibility issues as not all browsers support SharedArrayBuffer and makes it messy to use. Also multi-thread using massive memory (like 2 GB) that might not be affordable for low-end devices.

To improve performance without multi-threading, right now the most promising way is to use SIMD intrinsics, this is not easy but key to success. As a beginning, I would like to try it on x264 as mp4 is one of the most popular formats as it is.

• Speed up x264 with SIMD intrinsics
• Speed up ffmpeg with SIMD intrinsics
• Speed up other codecs if possible

Features

I know there are lots of features requested by the community, and some of them (ex. RTSP) are queried many times. But the fact is some features actually limited by the current status of WebAssembly which makes it almost technically impossible at the moment. Thus it requires extra amount of efforts to figure out a workaround or even changes in WebAssembly itself. I think features like this should be discussed in an issue or thread for a long time, and requires everyone's contribution to make it true.

• Accumulate long term discussions in Github Discussion
• Upgrade to ffmpeg n5.1

---

I believe there are much more we can and should improve here, but for now I would like to focus on these aspects / tasks and see how it goes. I cannot guarantee how long it takes for issues like performance, but definitely I am happy to share with everyone when there is any progress. At the same time it is always welcome to submit PRs or just provide your ideas. 😄

[milkliker]

So exciting to find Speed up x264 with SIMD intrinsics 🎊

[milkliker]

Do we have a plan of upgrade ffmpeg to 5.x or 4.4.x ?

[nuthinking]

I can confirm, in 4.4 the washed-out issue is fixed.

[jeromewu]

In that case, you might need to wait the upgrade or build it yourself.

[nuthinking]

I'm not in a rush, is there a rough ETA? 😇 Thanks!

[jeromewu]

Hard to say, maybe around end of Oct.

[nuthinking]

That would be great!

[jeremytenjo]

Thank you for working so hard on this project! I love it.

[Offroaders123]

This is a great project, nice work! Going to use it for a new project of mine.
Are there any plans to add ESM support? It's fine not to have it, but it has been really nice to use for projects that do have support for it. Thank you!
*Edit: In regards to using it in the browser (Node would work too, but the browser is the main benefit that is cool to use).

[jeromewu]

ESM support is included in the next version. 😄

[codenlighten]

great job brother.

[azickri]

Is it possible, if later ffmpeg.wasm can run without SharedArrayBuffer?
Because in version 113 of Google Chrome, it can no longer be used.
Because, if I add a Header like

• Cross-Origin-Embedder-Policy: require-corp
• Cross-Origin-Opener-Policy: same-origin
• Cross-Origin-Resource-Policy: same-origin
  It will block other requests, such as Media, API, and others.

Article: https://developer.chrome.com/blog/enabling-shared-array-buffer/#cross-origin-isolation

[ButzYung]

Because in version 113 of Google Chrome, it can no longer be used.

Where did you get this info from? Chrome 111 still have new feature related to SharedArrayBuffer and it doesn't make sense to me that they will shut it down 2 versions later.

https://chromestatus.com/feature/4668361878274048

PS: Personally I would like to see a version of ffmpeg.wasm running without SharedArrayBuffer though.

[azickri]

This is what I read from the article above.
Or, maybe I misunderstood the information from the article?

[tini2n]

my + on this

[tenthirtyone]

You need to fix your memory allocation problems. That should be the only item on the roadmap.

[AGPX]

I can't agree more: in my web video editor, I run ffmpeg wasm but the memory it consumes every time it never releases!

[zhengcling]

To improve performance without multi-threading, right now the most promising way is to use SIMD intrinsics, this is not easy but key to success. As a beginning, I would like to try it on x264 as mp4 is one of the most popular formats as it is.

When will this version be released? Thanks! @jeromewu

[quink-black]

I have POC implementation of FFmpeg HEVC IDCT with WASM simd128. https://ffmpeg.org/pipermail/ffmpeg-devel/2024-November/336009.html

The result looks not bad:

Benchmark with wasmtime:
wasmtime checkasm --test=hevc_idct --benchmark

hevc_idct_4x4_8_c: 20.8 ( 1.00x)
hevc_idct_4x4_8_simd128: 10.2 ( 2.02x)
hevc_idct_4x4_10_c: 21.0 ( 1.00x)
hevc_idct_4x4_10_simd128: 10.2 ( 2.05x)
hevc_idct_8x8_8_c: 198.0 ( 1.00x)
hevc_idct_8x8_8_simd128: 41.5 ( 4.77x)
hevc_idct_8x8_10_c: 197.8 ( 1.00x)
hevc_idct_8x8_10_simd128: 41.8 ( 4.74x)
hevc_idct_16x16_8_c: 1385.2 ( 1.00x)
hevc_idct_16x16_8_simd128: 291.8 ( 4.75x)
hevc_idct_16x16_10_c: 1375.0 ( 1.00x)
hevc_idct_16x16_10_simd128: 281.2 ( 4.89x)
hevc_idct_32x32_8_c: 13479.0 ( 1.00x)
hevc_idct_32x32_8_simd128: 1541.5 ( 8.74x)
hevc_idct_32x32_10_c: 13448.0 ( 1.00x)
hevc_idct_32x32_10_simd128: 1541.5 ( 8.72x)

Decoding test:

1. Without the optimization:
   $ wasm ./ffmpeg -cpuflags 0 -i 265.mp4 -an -t 120 -f null - -benchmark
   frame= 2880 fps= 94 q=-0.0 Lsize=N/A time=00:02:00.00 bitrate=N/A speed=3.92x

2. With the optimization:
   $ wasm ./ffmpeg -i 265.mp4 -an -t 120 -f null - -benchmark
   frame= 2880 fps=110 q=-0.0 Lsize=N/A time=00:02:00.00 bitrate=N/A speed=4.58x

But FFmpeg require handwritten assembly than intrinsics normally. I have no insterest to write handwritten assembly in WASM (serious?). Not sure will the work go any further. Let's see.