Unblock AudioWorklets: Find an alternative to TextEncoder / TextDecoder

[kettle11]

AudioWorklet's are a great fit for WebAssembly and therefor they're a great fit for Rust. wasm-bindgen is almost ready to be used in AudioWorklets, but not quite.

As discussed in a previous issue (rustwasm/wasm-pack#689) the major blocker is that TextEncoder and TextDecoder are not available within AudioWorklets. wasm-bindgen would need to provide an alternative or find a work around.

---

In the above linked issue @cconnection made a prototype using a polyfill for TextEncoder / TextDecoder: https://github.com/anonyco/FastestSmallestTextEncoderDecoder

wasm-bindgen could potentially incorporate such a polyfill and then have some sort of "AudioWorklet" mode where it uses such a polyfill.

Or perhaps there are other ideas?

[alexcrichton]

For this sort of functionality we typically rely on a bundler, do you know if bundlers are capable of injecting an implementation of these types?

[BenoitDel]

Webpack uses async import for its wasm files. The old sync (webpack 4) import is deprecated, and replace by the ES Module Integration Proposal which is async.

Audioworklet context support only static import (not supported in FF, only in Chrome for now), the default wasm import from webpack 5 is not available in this context.

I tried also target web and target no-modules using init function, but it doesn't work either cause fetch is not supported in audioworklet context.

The only solution i found is to retrieve the wasm module in the main thread via a fetch, then send the module via message.postMessage().

I also know that emscripten is able to be imported directly in audioworklet see example. But i am not sufficiently competent to understand how they do it.

Maybe we could try to make a working example for everyone, and see from there if there is a necessity to add a new target to wasm-pack ?
I can try to write something (even if i am beginner) ?

[BenoitDel]

I find another way to import wasm code inside audioworklet context by using processorOptions (specs).
The data is available via options inside the audioworkletnode.

class WhiteNoiseProcessor extends AudioWorkletProcessor {
  constructor(options) {
    super();
    const bufferSrc = new Uint8Array(options.processorOptions.data);
    WebAssembly.instantiate(bufferSrc, {}).then((res) =>
      console.log("res: ", res.instance.exports)
    );
  }

And in rust, you do it like that:

const WASM: &[u8] = include_bytes!("processor_bg.wasm");
        let data_proc = ProcData { data: WASM };
        let mut options = AudioWorkletNodeOptions::new();
        options.processor_options(Some(&Object::from(
            JsValue::from_serde(&data_proc).unwrap(),
        )));
        let worklet_node = AudioWorkletNode::new_with_options(&self.ctx, "noise", &options)?;

It works at least in Chrome 87 and FF 83

[alexcrichton]

@BenoitDel I think you might have a somewhat separate problem about instantiation whereas this issue is more about even if we fixed instantiation there's still the Text{Encoder,Decoder} problem.

If webpack and other bundlers don't really support this then adding a new target to wasm-bindgen seems fine to me.

[BenoitDel]

@alexcrichton Yes you are right, i'm off topic.
is it a good idea to give access to TextEncoder inside a high priority thread like an audioworklet?

[alexcrichton]

Unfortunately I don't have the answer to questions like that, I don't even know what audio worklets are.

[kettle11]

@alexcrichton Audio Worklets are a way to run code on the browser's dedicated audio thread.

I'm specifically interested in them as they may allow sharing high-performance / low-latency audio code between desktop and web.

By design what you can do within them is very constrained because if the audio thread lags the user may hear artifacts. Unfortunately those constraints also make them hard to work with in our case.

[thomaskvnze]

@kettle11 as you mentioned, the audio thread should focus on processing audio which is normally represented as floats. This is why there is nothing like text encoding / decoding because you don't need string data types in audio processing. And you should also not do any text encoding in the audio thread in order to be as performant as possible. So I ditched the idea to use anything expect integers and floats in my rust functions which are called by javascript. If you want more readable code in your rust code base, you can use consts / enums to give your floats / ints some semantic meaning.

[ColinTimBarndt]

@cconnection There is always at least text decoding because of the error handling:

imports.wbg.__wbindgen_throw = function(arg0, arg1) {
    throw new Error(getStringFromWasm0(arg0, arg1));
};

Is there any way to prevent this from being generated?

[kettle11]

I've been looking into this a bit more again.

An approach to getting around the missing text decoder is described here: https://www.toptal.com/webassembly/webassembly-rust-tutorial-web-audio

The Audio Worklet can use a polyfill (this is the one from the above article: https://gist.github.com/Yaffle/5458286) to implement a TextDecoder that wasm-bindgen can use.

There are obviously many APIs Audio Worklets cannot access (like Window) but perhaps it's ok if those simply error when they're called.

Looking beyond that another issue is that wasm-bindgen should probably have a way to use Audio Worklets (and Workers) in a way more similar to standard threads. It should be possible for a binary with main to spawn an Audio Worklet and that Audio Worklet should be able to call an arbitrary exported Rust function as an entry point instead of also calling into main.

Presently __wbindgen_start detects if it's called off the main thread, sets up a stack, and then calls main. It'd be nice if setting up the stack were decoupled so that a custom entry-point can be used.

[kettle11]

This isn't exactly on topic, but perhaps useful to others trying to do similar work:

I managed to cobble together some code that works with Audio Worklets, Rust, and partially wasm-bindgen and put it into a weird demo here (only works in Chrome right now): https://kettlecorn.itch.io/rusty-audio-test

Two hacks were needed

• To correctly initialize the stack pointer I had to modify wasm-bindgen to export a function that initializes the stack without calling main. I'd create a pull-request here, but I think a feature that allows alternative entry-points needs more thought than my quick hack.

• In the AudioWorklet I wrote code that replaced all wasm-bindgen imports with stubs. This is a heavy-handed hack, but it's probably workable in many cases because AudioWorklet code will rarely call out to Javascript anyways. It does really mess up logging and panics though. It looks like this:

imports.wbg = {};
WebAssembly.Module.imports(e.data.wasm_module).forEach(item => {
    if (imports[item.module] === undefined) {
        imports[item.module] = {};
    }
    if (item.kind == "function") {
        imports[item.module][item.name] = function () {
            console.log(item.name + "is unimplemented in audio worklet");
        }
    }
    if (item.kind == "memory") {
        imports[item.module][item.name] = {};
    }
});
// Import the memory that's passed in a message from the main thread.
imports.wbg.memory = e.data.wasm_memory;

The combo of these two hacks allowed the AudioWorklet code to work fine alongside other Rust / wasm-bindgen running in the main thread.

[lukaslihotzki]

A Rust crate can directly polyfill TextEncoder and TextDecoder in the resulting wasm_bindgen JS module: https://github.com/rustwasm/wasm-bindgen/pull/3017/files#diff-c84654e0d49a636c151e15aeff24bc4374ceb228afa336a25a4cbdf41c6690c2R54-R60

I don't know if this is better or worse than the other workarounds suggested here, but this way, no extra scripts or bundler setup is required by the application developer. Just include the library in Cargo.toml, and it will work, even with WASM imports inside the worklet.

[jamsinclair]

Apologies for reviving an older thread.

Seems like it didn't come up in the discussion, but another situation where TextEncoder / TextDecoder apis are not available occurs when running wasm glue code in standalone V8 JavaScript engine environments. Such as Cloudflare Workers , Vercel Edge Functions and other similar services. Only a limited subset of JS APIs are exposed and implemented.

If anyone stumbles upon this thread and were using those services, note that you'll want to polyfill TextEncoder/TextDecoder as noted in other comments above 🤓