Audio Worklets #16449

juj · 2022-03-08T10:01:03Z

Add support for Wasm Audio Worklets, based on Wasm Workers.

Wasm Audio Worklets enables code to create Wasm-based Web Audio processing nodes, that run on a dedicated thread of their own. (all such nodes run sequentially on a single main audio thread, there won't be a single thread for each node)

Fix Chrome not continuing the AudioWorklet processing if a number 1 is returned from the callback - must return 'true' specifically. Add new tone generator sample. Adjust comment. Use MessagePort.onmessage instead of add/removeEventListener(), since onmessage .start()s the MessagePort automatically. Fix name noise-generator to tone-generator Improve assertions.

…IS_AUDIO_WORKLET.

# Conflicts: # emcc.py # src/settings.js

…depended on.

juj · 2022-03-16T11:44:04Z

This PR has now been merged on top of latest main, and is good for final review and landing.

src/audio_worklet.js

kripken · 2022-03-25T16:17:24Z

What do you think about adding a doc page with an overview of the API and some simple examples? I think that would be useful in general for users later but also help in review.

How does this relate to WebAudio? I see webaudio.h also has methods related to that and not to audio worklets. I'm not very familiar with either WebAudio or AudioWorklets however so maybe that's something obvious, apologies if so.

Also please add a changelog entry.

kripken · 2022-03-25T16:21:34Z

cc @hoch , this might interest you (no worries if you don't have time, of course).

tklajnscek · 2022-04-06T08:38:30Z

Hey @juj!

Finally got around to testing this out along with the general wasm workers support. Thanks for all the work on this!

The first realization I had was that with pure wasm workers none of the standard library code will work multithreaded (e.g. malloc will not be thread-safe etc) and we have quite a bit of code (from 3rd parties as well) that would need to change so for the foreseeable future pthreads is what we're stuck with.

Looking through the tests I realized we can run pthreads and wasm workers side by side so I set out to test running just the audio worklet bit as a wasm worker using this the code from this PR, but with pthreads enabled the JS doesn't load in an audio worklet context.

First it asserts, but removing that it tries running PThread.init and fails etc.

So I went and took this PR's branch, rebased it on latest main for ease of testing and made a few changes to skip over any remains of PThread code when not running in a PThread context.

Here's the commit - I can't do an automatically mergable PR to this branch because it's on a rebased branch:
tklajnscek@1824e25

This makes the simple example work.

I'll run some more complex setup and update here.

tklajnscek · 2022-04-11T08:44:16Z

Here's quick status update...

I was able to make it work with a combination of pthreads for normal threads and wasm workers for the audio worklet code.

I used the synchronization primitives from wasm_worker.h to setup the communication between the audio worklet and the pthreads and it worked fine.

In this specific use case I had to get a middleware product ready for production so I actually trimmed down the worklet code so much that in the end the pthread part uses wasm_worker sync primitives in C++ code compiled to wasm like you'd expect, but the audio worklet part is pure javascript that uses Atomics.notify etc directly and it works fine and avoids a dependency on this PR in our deployment :)

I will schedule some time in the following weeks to look at our full blown audio playback engine in the context of this PR and to see what it would take to make the audio worklet code not depend on pthreads at all, but have some other priorities I need to deal with first (like always) :)

juj · 2022-04-12T18:06:51Z

What do you think about adding a doc page with an overview of the API and some simple examples? I think that would be useful in general for users later but also help in review.

Sure, I'll add a doc page. The added tests serve as simple examples, so pointing users to them to get started.

How does this relate to WebAudio? I see webaudio.h also has methods related to that and not to audio worklets. I'm not very familiar with either WebAudio or AudioWorklets however so maybe that's something obvious, apologies if so.

Audio Worklets are a subpart of Web Audio API. So in order to enable integration with Audio Worklets, we need to have at least a minimal integration/understanding of Web Audio AudioContext object with the PR. That is why I added a shallow shim that allows creating an AudioContext from C side, and then a C&JS mechanism to manage the Wasm AudioContext<->int opaque handle binding.

Also please add a changelog entry.

Will do, I'll address review once I get through my next round of bug PRs.

…TS_GLOBALTHIS internal setting to help simplify this.

… runs.

…ore#18402.

juj · 2023-02-02T19:40:44Z

I noticed when merging with main that there were some regressions with compatibility against main. Updated with fixes to those, and fixed the test suites to work again. Would be be closer towards a merge? :)

kripken · 2023-02-02T20:41:43Z

emcc.py

@@ -2352,6 +2352,19 @@ def phase_linker_setup(options, state, newargs):
      settings.WASM_WORKER_FILE = unsuffixed(os.path.basename(target)) + '.ww.js'
    settings.JS_LIBRARIES.append((0, shared.path_from_root('src', 'library_wasm_worker.js')))

+  settings.SUPPORTS_GLOBALTHIS = settings.MIN_CHROME_VERSION >= 71 and settings.MIN_EDGE_VERSION >= 79 and settings.MIN_FIREFOX_VERSION >= 65 and settings.MIN_IE_VERSION == settings.TARGET_NOT_SUPPORTED and settings.MIN_SAFARI_VERSION >= 120100 # and settings.MIN_NODE_VERSION >= 120000


Rather than baking this into emcc.py, see

https://github.com/emscripten-core/emscripten/blob/main/tools/feature_matrix.py

The numbers can be in there and this place can call caniuse().

kripken · 2023-02-02T20:42:47Z

Lgtm aside from the caniuse issue and the discussion around the comparison to ENVIRONMENT_IS_PTHREAD (but that discussion doesn't matter much).

juj · 2023-02-03T11:34:17Z

Lgtm aside from the caniuse issue and the discussion around the comparison to ENVIRONMENT_IS_PTHREAD (but that discussion doesn't matter much).

Updated code to utilize caniuse. On the ENVIRONMENT_IS_PTHREAD / ENVIRONMENT_IS_AUDIO_WORKLET part, I think the current code form is required.

… IN_TEST_HARNESS increased in size (added 'typeof ENVIRONMENT_IS_AUDIO_WORKLET&&ENVIRONMENT_IS_AUDIO_WORKLET)' code)

kripken

lgtm!

Great to have this functionality, I know many have been waiting for it.

Jonathhhan · 2023-02-16T01:50:26Z

Thank you very much for implementing this. :)

This commit changes the old term `readAsmConstArgs` to the new term `readEmAsmArgs`, due to conflict of emscripten-core#16449 and emscripten-core#18218 Background history: 1. emscripten-core#16449 proposed, introducing AudioWorklet earlier 2. emscripten-core#18218 proposed, renaming readAsmConstArgs to readEmAsmArgs 3. emscripten-core#18218 merged earlier 4. emscripten-core#16449 merged later, using the old term `readAsmConstArgs`

It fixes emscripten_audio_worklet_post_function_sig, which was previously not worked, and enhances test to cover all emscripten_audio_worklet_post_function_*. In order to fix, it changes the old term `readAsmConstArgs` to the new term `readEmAsmArgs`, due to conflict of emscripten-core#16449 and emscripten-core#18218 Background history: 1. emscripten-core#16449 proposed, introducing AudioWorklet earlier 2. emscripten-core#18218 proposed, renaming readAsmConstArgs to readEmAsmArgs 3. emscripten-core#18218 merged earlier 4. emscripten-core#16449 merged later, using the old term `readAsmConstArgs` enhance the test for cover all emscripten_audio_worklet_post_function_*

This was referenced Mar 8, 2022

Wasm workers #12833

Merged

Basic support for running Emscripten code as AudioWorklets #12502

Closed

juj added 2 commits March 8, 2022 12:52

Add src/audio_worklet.js to eslint ignore

d2d9176

Optimize code size

10b7322

Jonathhhan mentioned this pull request Mar 11, 2022

OF 0.12.0 Release Checklist openframeworks/openFrameworks#6692

Closed

77 tasks

juj added 15 commits March 11, 2022 12:20

Add emscripten_current_thread_is_audio_worklet(), remove ENVIRONMENT_…

de13987

…IS_AUDIO_WORKLET.

Fix to work with Closure

194896b

Simplify MINIMAL_RUNTIME shell module preamble generation.

cfca4a5

Fix Closure and simple AudioWorklet creation

75a744d

Fix Module import for AudioWorklets, and move towards globalThis

1105580

Disable -sAUDIO_WORKLET + -sTEXTDECODER=2 combo

59475e5

Fix shell case

2c0dacb

Mark AUDIO_WORKLET and SINGLE_FILE not mutually compatible

397820c

Default runtime support

9c12ec3

Merge remote-tracking branch 'origin/main' into audio_worklets

0a5cc67

# Conflicts: # emcc.py # src/settings.js

Revert unnecessary code

629f7b9

Fix merge

c666d74

AudioWorklets functioning in default runtime.

674469d

Remove #if WASM_WORKERS checks since Wasm Workers is unconditionally …

7b56e06

…depended on.

Add more interactive tests

26bd5b6

Test Audio Worklets with Closure

666d657

juj enabled auto-merge (squash) March 16, 2022 14:26

msabwat reviewed Mar 23, 2022

View reviewed changes

src/audio_worklet.js Show resolved Hide resolved

juj added 9 commits February 1, 2023 16:57

Fix MINIMAL_RUNTIME test run

5bbfa02

Fix tests

d5d5050

Fix test

072a41f

Remove duplicated part

e7e8b24

Clean up shell_minimal.js handling of Module import, introduce SUPPOR…

ae19a6e

…TS_GLOBALTHIS internal setting to help simplify this.

Fix browser test harness to take into account errors in audio worklet…

3a46c1a

… runs.

Fix Audio Worklet futex waits to work after change in PR emscripten-c…

aac666c

…ore#18402.

Fix typo

81b1fd8

Fix Audio Worklets stack exporting after upstream change.

db20434

kripken reviewed Feb 2, 2023

View reviewed changes

Move check for globalThis into feature_matrix.py

3b31c2b

juj added 3 commits February 3, 2023 13:34

Flake

c871919

Merge remote-tracking branch 'origin/main' into audio_worklets

b494b1e

Allow more slack in test_small_js_flags because the scaffold code for…

c16c298

… IN_TEST_HARNESS increased in size (added 'typeof ENVIRONMENT_IS_AUDIO_WORKLET&&ENVIRONMENT_IS_AUDIO_WORKLET)' code)

kripken approved these changes Feb 3, 2023

View reviewed changes

juj added 2 commits February 3, 2023 22:16

Merge remote-tracking branch 'origin/main' into audio_worklets

605319e

Merge remote-tracking branch 'origin/main' into audio_worklets

1d2df94

juj force-pushed the audio_worklets branch from ca057cd to 1d2df94 Compare February 5, 2023 08:22

juj merged commit 5402fc9 into emscripten-core:main Feb 5, 2023

mighty1231 mentioned this pull request Dec 9, 2024

[AUDIO_WORKLET] Fix emscripten_audio_worklet_post_function_sig #23108

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Audio Worklets #16449

Audio Worklets #16449

Uh oh!

juj commented Mar 8, 2022

Uh oh!

juj commented Mar 16, 2022

Uh oh!

Uh oh!

kripken commented Mar 25, 2022

Uh oh!

kripken commented Mar 25, 2022

Uh oh!

tklajnscek commented Apr 6, 2022

Uh oh!

tklajnscek commented Apr 11, 2022

Uh oh!

juj commented Apr 12, 2022

Uh oh!

juj commented Feb 2, 2023

Uh oh!

kripken Feb 2, 2023

Uh oh!

juj Feb 3, 2023

Uh oh!

kripken commented Feb 2, 2023

Uh oh!

juj commented Feb 3, 2023

Uh oh!

kripken left a comment

Uh oh!

Jonathhhan commented Feb 16, 2023

Uh oh!

Uh oh!

Audio Worklets #16449

Audio Worklets #16449

Uh oh!

Conversation

juj commented Mar 8, 2022

Uh oh!

juj commented Mar 16, 2022

Uh oh!

Uh oh!

kripken commented Mar 25, 2022

Uh oh!

kripken commented Mar 25, 2022

Uh oh!

tklajnscek commented Apr 6, 2022

Uh oh!

tklajnscek commented Apr 11, 2022

Uh oh!

juj commented Apr 12, 2022

Uh oh!

juj commented Feb 2, 2023

Uh oh!

kripken Feb 2, 2023

Choose a reason for hiding this comment

Uh oh!

juj Feb 3, 2023

Choose a reason for hiding this comment

Uh oh!

kripken commented Feb 2, 2023

Uh oh!

juj commented Feb 3, 2023

Uh oh!

kripken left a comment

Choose a reason for hiding this comment

Uh oh!

Jonathhhan commented Feb 16, 2023

Uh oh!

Uh oh!