Repository navigation
[FEAT] Feature Request: Support for the Unity Engine #62
Description
Activity
You know, after more than six years of deep diving into Unity, I've honestly never hit a wall with its built-in
AudioSourcecomponents and effects. Because of that, I really struggle to see a compelling use for something extra like SoundFlow in Unity, unless maybe you're building some kind of dedicated audio editing suite. Even then, frankly, Unity might introduce more overhead than it's worth for that specific task.My real motivation for building SoundFlow was actually to fill a gap I saw in the broader .NET ecosystem – specifically, getting truly cross-platform audio without drowning in dependency hell. But when it comes to Unity, its audio ecosystem already feels pretty solid. You've got the native components, tons of external packages, and even professional tools like FMOD available.
So, that's why I'm genuinely curious, what unique value or functionality do you envision SoundFlow bringing to the Unity development environment?
What I might need is the ability to process microphone recordings using WebRTC audio processing, which Unity doesn't natively implement. I'd like to play around with Sound Flow more to make a better recording tool plugin, including comprehensive audio processing capabilities.
I'm not saying this won't be implemented, but if it's a critical need, you can integrate the WebRTC APM extension directly from source file AudioProcessingModule.cs without depending on the full SoundFlow library. This would primarily involve removing the
ISoundModifierinterface and adapting theAudioEngine.OnAudioProcessed += HandleAudioEngineProcessedForAec;callback to work directly with Unity'sMicrophone.Start(...)API. Otherwise, I'll address this after feature requests #26 and #60 have been completed.Got it, thanks for the pointer. I'll give it a shot and try to implement it with AudioProcessingModule.cs.
Reacted by xuefeiContinuously monitor this issue, Unity's DLL dependencies are like hell……
While I was preparing to address that, I remembered that Unity's latest version was using C# 7.3 or 8.0 back then. I've now found they've updated to C# 9.0. My library is using C# 12.0, and I am planning to upgrade to C# 14.0 when a stable LTS version of .NET 10.0 is released. Therefore, I have no plans to downgrade the language version, which has presented a roadblock for me.
Is there a way to run this without disrupting the language version and syntax? Would providing a compiled DLL version tackle that? I haven't worked with compiled libraries in Unity before; I've always loved to keep things in their source shape.
While I was preparing to address that, I remembered that Unity's latest version was using C# 7.3 or 8.0 back then. I've now found they've updated to C# 9.0. My library is using C# 12.0, and I am planning to upgrade to C# 14.0 when a stable LTS version of .NET 10.0 is released. Therefore, I have no plans to downgrade the language version, which has presented a roadblock for me.
Is there a way to run this without disrupting the language version and syntax? Would providing a compiled DLL version tackle that? I haven't worked with compiled libraries in Unity before; I've always loved to keep things in their source shape.
Um, your workload is a bit too much. Moreover, the problem is not entirely due to the downgrade of the C# version. Unity's dll dependency is a difficult problem to solve. I modified some codes from your project and put it into Unity to achieve noise reduction for wav files. It sounds good.
https://github.com/xue-fei/soundflow-unity/blob/master/Assets/soundflow-unity/Test.csI migrated part of your project code to Unity. After testing, the noise reduction and recording to files work normally, but there is no sound when playing WAV files. I tried to switch the output device, but it still doesn't work. I don't know why...
25.7.7 Unity now is ok……
I migrated part of your project code to Unity. After testing, the noise reduction and recording to files work normally, but there is no sound when playing WAV files. I tried to switch the output device, but it still doesn't work. I don't know why...
25.7.7 Unity now is ok……
Unity working now? what was the issue, a native or managed code?
I migrated part of your project code to Unity. After testing, the noise reduction and recording to files work normally, but there is no sound when playing WAV files. I tried to switch the output device, but it still doesn't work. I don't know why...
https://github.com/xue-fei/soundflow-unity/blob/master/Assets/soundflow-unity/Samples/SimplePlayer/UnitySimplePlayer.cs
25.7.7 Unity now is ok……Unity working now? what was the issue, a native or managed code?
There is no problem running in Unity. How can I obtain the audio data after real-time noise reduction or echo cancellation?
There is no problem running in Unity. How can I obtain the audio data after real-time noise reduction or echo cancellation?
Without modifying the library code, simply create an analyzer and add it to the sound player. If that's not possible, then modify the APM modifier to fire an event with the cleaned audio.
For example:
using SoundFlow.Abstracts; using SoundFlow.Interfaces; using System; namespace SoundFlow.Analyzers; public class CallbackAnalyzer : AudioAnalyzer { /// <inheritdoc /> public override string Name { get; set; } = "Callback Analyzer"; /// <summary> /// Event that is raised when audio data has been analyzed. /// Subscribers will receive a read-only span of the audio buffer. /// </summary> public event Action<ReadOnlySpan<float>>? AudioAvailable; /// <summary> /// Initializes a new instance of the <see cref="CallbackAnalyzer"/> class. /// Note: This analyzer does not use the IVisualizer, so it is ignored. /// </summary> public CallbackAnalyzer() : base(null) { } /// <summary> /// Raises the AudioAvailable event, passing the audio buffer to any subscribers. /// </summary> /// <param name="buffer">The audio buffer to be passed to subscribers.</param> protected override void Analyze(Span<float> buffer) { // Raise the event, notifying any subscribers and passing them the data. // We pass it as a ReadOnlySpan to prevent subscribers from modifying the original buffer. AudioAvailable?.Invoke(buffer); } }Thank you very much. The above code can indeed obtain the processed audio data. I tested that the echo cancellation is indeed effective. I am currently working on a digital human project. The problem I encountered is that when the digital human is speaking, it will affect the wake-up effect. In other words, the sound of the digital human speaking and my wake-up word sound enter the microphone data together. Can SoundFlow remove the digital human's voice from the microphone data?
Apologies for the delayed response; I had mistakenly thought I had already responded to this via email, perhaps a different mail.
Can SoundFlow remove the digital human's voice from the microphone data?
Regarding the SoundFlow APM extension, it is built upon WebRTC, specifically leveraging its integrated Acoustic Echo Cancellation (AEC) module. The AEC's function is to identify and then remove sounds that are essentially "echoes" – audio that has been output and subsequently re-captured (e.g., from a speaker into a microphone).
The challenge with a digital human's initial speech is that it's not an echo in the traditional sense; it's the primary sound output. If the AEC attempts to "memorize" and cancel this initial output, it's effectively treating the original source as a feedback loop. Handling this specific type of internal "feedback" (where the original, non-echoing speech is cancelled) falls outside the design scope of WebRTC's AEC and, by extension, SoundFlow's capabilities.
Reacted by xuefeiYes, I understand this question, thank you for your multiple replies.
Requirements
1. Is your feature request related to a problem? Please describe.
When the SoundFlow.dll (compiled for .NET 8.0) is imported into a Unity project, the following two critical assembly resolution errors occur, preventing the library from loading:
These errors are caused by direct dependencies on two APIs that are not fully compatible with Unity's .NET runtime environment:
System.Runtime.Intrinsics: Used extensively in SoundFlow.Utils.MathHelper for hardware-accelerated (SIMD) calculations for FFT and windowing functions.
System.Text.Json: Used for JSON serialization tasks. While modern Unity versions have improved support, it is not universally available or is often less preferred than established solutions within the ecosystem.
2. Describe the Solution You'd Like
The proposed solution is to refactor the library to abstract these specific dependencies behind interfaces, allowing different implementations to be injected based on the host platform.
Action: Create an IMathOperations interface for methods that currently use SSE/AVX (e.g., Fft, HammingWindow).
Default Implementation: A NetCoreIntrinsicsProvider class will contain the existing, high-performance SSE/AVX code.
Unity Implementation: A UnityBurstProvider class will implement the same interface using Unity's Burst compiler and Unity.Burst.Intrinsics for equivalent, high-performance SIMD operations. A fallback to the existing scalar logic can be used for non-Burst scenarios.
Action: Create an IJsonSerializer interface with Serialize(T obj) and Deserialize(string json) methods.
Default Implementation: A SystemTextJsonProvider class will use System.Text.Json to handle serialization for standard .NET environments.
Unity Implementation: A UnityJsonProvider class will implement the interface using a Unity-compatible solution, such as UnityEngine.JsonUtility (for simple cases) or a third-party library like Newtonsoft.Json (which is standard in many Unity projects).
This dual-interface approach resolves all compatibility errors while making the entire library more modular, maintainable, and extensible.
3. Describe Alternatives You've Considered
The only alternative is a manual fork and rewrite of the library's internals, which is unsustainable. It would prevent users from receiving updates and would fragment the codebase. The proposed interface-based solution is the clean way to support multiple platforms.
4. Proposed API (if applicable)
The core idea is to use dependency injection to provide the correct implementation at runtime.
Global Configuration:
Example JSON Interface:
This API design is clean, flexible, and completely decouples the core library logic from the platform-specific implementations.
5. Benefits
Full Unity Compatibility: Directly solves the blocking issues, making SoundFlow available to a vast new user base.
Enhanced Extensibility: Developers are no longer locked into System.Text.Json. They could, for instance, create their own NewtonsoftJsonProvider if they prefer its features, even in a standard .NET project.
Future-Proof Architecture: This modular design makes it trivial to add support for other platforms (e.g., MAUI, Godot) in the future by simply adding a new implementation of the interfaces.
Performance without Compromise: The use of Burst for math operations ensures that Unity developers still get the best possible performance.
6. Potential Drawbacks/Challenges
Initial Refactoring Effort: There is an upfront cost to refactoring MathHelper and any classes that use System.Text.Json.
Feature Parity: If the library uses advanced features of System.Text.Json (e.g., custom converters, reference handling), care must be taken to ensure the chosen Unity-compatible serializer (JsonUtility or Newtonsoft.Json) can support them.
Build/Distribution Strategy: A clear strategy for packaging the core library and the platform-specific provider packages will be needed (e.g., via separate NuGet packages or Unity packages).
7. Additional Context
The provided code for MathHelper is a clear example of where the System.Runtime.Intrinsics abstraction is needed. A similar pattern of direct dependency likely exists for System.Text.Json and would benefit equally from this proposed refactoring. Applying this strategy to both dependencies is the key to creating a truly portable and more robust SoundFlow library.