Audio Filters and Digital Signal Processors

[harudagondi]

What problem does this solve or what need does it fill?

Imagine you are in a water level, but still above ground. The music plays normally.

Then you go underwater, thus the music becomes all muddied up.

Next, you jump out of the water, and the music returns to normal.

How do you implement that in Bevy?

• Godot has audio buses.
• Unreal Engine 5 has MetaSounds.
• Unity has audio effects.

What solution would you like?

An AudioFilter trait, with a single method filter(&mut self, input: &[Sample], output: &mut [Sample]), where Sample is simply a type that implements rodio::Sample.

Boxed audio filters are stored in Audio, in order, and each can be toggled programmatically. These effects are global (unless #5832 is implemented, then it is local to each AudioListener).

What alternative(s) have you considered?

Implement an audio filter using Decodable and pass the audio source to it. Similar to iterator combinators or fundsp's audio units combinators. Very unergonomic when manually implemented.

Additional context

N/A

[DevinLeamy]

Hi @harudagondi, I'm interested in this issue and harudagondi/bevy_oddio#31 and have some questions.

What is the reasoning behind having filter take in input and output slices, as opposed to something like filter(&mut self, input: &Sample) -> Sample?

Based on what you said, I imagined an API that looks something like

impl AudioFilter for AmplifyFilter {
    fn filter(&self, sample: &Sample) -> Sample {
        --snip--
    }
}

impl AudioFilter for DistortionFilter {
    fn filter(&self, sample: &Sample) -> Sample {
        --snip--
    }
}

fn setup(audio: ResMut<Audio>, ...) {
    let amplify = AmplifyFilter::new();
    let distort = DistortionFilter::new();

    audio.insert_filter("Amplify", amplify);
    audio.insert_filter("Distort", distort);

    --snip--

    audio.play(music);
}

fn toggle_filter_system(audio: ResMut<Audio>, ...) {

    --snip--

    if (toggle_distortion) {
        audio.toggle_filter("Distort");
        /*
             .enable_filter(...);
             .disable_filter(...);
             .remove_filter(...);
        */
    }
}

How does this design look? And, did you have any idea as to how the user should define filters for samples? My initial idea was to have something similar to rodios decorator pattern like Sources, but for Samples. Something like sample.amplify(...).fade_in(...). I'd like to get your thoughts.

If there's anything else I should now about this issue, lmk!

[harudagondi]

What is the reasoning behind having filter take in input and output slices, as opposed to something like filter(&mut self, input: &Sample) -> Sample?

This is honestly just arbitrary. Kira's Sound returns a Frame, while oddio's Signal take in mutable output reference. I don't know the exact reasoning for the API.

I agree with the toggle system, but for the setup, I imagine a node based system with one input and output. This allows for more flexibility (See fundsp's Net32/64, which is a DAG). But for a minimal viable product, this would suffice, although I would like to see a way to easily reorder these filters.

And, did you have any idea as to how the user should define filters for samples? My initial idea was to have something similar to rodios decorator pattern like Sources, but for Samples. Something like sample.amplify(...).fade_in(...). I'd like to get your thoughts.

I'm not sure I understand. Can you elaborate further on this?

[harudagondi]

Also, if you'd like for us to informally discuss this, you could say hi at the #audio-dev channel at the bevy discord 😄

[DevinLeamy]

(See fundsp's Net32/64, which is a DAG)

This is awesome and something like this for DSP and filtering would be awesome!

[harudagondi]

By the way, there are talks in bevy development that there would be a possibility to have a bevy_graph crate similar to petgraph. We could utilize that for our audio node system.

We could also adopt how shaders work in rendering, however I have no experience in that field.

[adsick]

just want to let you know, there is https://github.com/WeirdConstructor/synfx-dsp-jit a dsp crate that is using cranelift to jit compile an ast for efficiency.

[adsick]

also want to throw a couple of pennies here:

1. think of naming, AudioFilter may be called AudioNode, AudioEffect, AudioProcessor etc. "filter()" could be "apply()", "process()" etc.

2. think of controlling this filters/fx whatever - you wanna have ability to control things like volume, mix (dry/wet), cutoff for filters, mix, reverb amount, size, delay length etc. these parameters will be different for each type of filter/fx

[harudagondi]

just want to let you know, there is WeirdConstructor/synfx-dsp-jit a dsp crate that is using cranelift to jit compile an ast for efficiency.

This is nice, however this is licensed as GPL-3.0-or-later. Also, this issue aims to find a way to introduce an interface for audio filters, not implementing them directly. However, that is an interesting project 😊.

[harudagondi]

think of naming, AudioFilter may be called AudioNode, AudioEffect, AudioProcessor etc. "filter()" could be "apply()", "process()" etc.

Yeah I agree with this.

think of controlling this filters/fx whatever - you wanna have ability to control things like volume, mix (dry/wet), cutoff for filters, mix, reverb amount, size, delay length etc. these parameters will be different for each type of filter/fx

Yeah this is also one of the things we would like to have. A related issue is #5828, of which we could apply to this one.

[adsick]

This is nice, however this is licensed as GPL-3.0-or-later. Also, this issue aims to find a way to introduce an interface for audio filters, not implementing them directly. However, that is an interesting project 😊.

yep, I've paid attention to that too. I did not meant we could use it but rather teach from it.