[QSTN] Streaming recorded audio with modifiers through Udp #111

Activity

1. atabaysagin

   added

   questionFurther information is requestedFurther information is requested

   on May 2, 2026
2. LSXPrime commented on May 11, 2026

   

   LSXPrime

   on May 11, 2026

   Owner

   More actions

   @atabaysagin Great question. I've actually shipped this exact workflow before using SoundFlow, To stream audio over a network in SoundFlow, you need to decouple the capture process from the playback process and connect them using a network socket (UdpClient in this case).

   1. The Sender (Broadcaster): We can use the Recorder component initialized with an AudioProcessCallback. We can add the WebRtcApmModifier directly to this Recorder. This way, the callback will yield the clean, processed audio buffer every frame, which we can cast to bytes and send via UDP.
   2. The Receiver (Player): We will use the QueueDataProvider. This is a special ISoundDataProvider designed specifically for pushing chunks of audio dynamically (e.g., from network packets). A SoundPlayer will read from this queue and play it through the AudioPlaybackDevice.

   But I'd strongly recommend skipping raw UDP in favor of Socket.IO or WebRTC DataChannels. Both integrate seamlessly with SoundFlow's existing Recorder → QueueDataProvider pipeline, and they solve the real-world headaches raw UDP brings (NAT traversal, packet reordering, encryption, and firewall blocking).

   The Pattern

   Mic → WebRtcApmModifier → Recorder callback → Opus encode → Emit via Socket.IO / DataChannel
                                                                    ↓
   Receive via Socket.IO / DataChannel → Opus decode → QueueDataProvider → SoundPlayer → Speakers
   

   Sender (Local)

   var recorder = new Recorder(captureDevice, (samples, cap) =>
   {
       // samples is already AEC+NS processed by WebRtcApmModifier
       var opusPacket = opusEncoder.Encode(samples, frameSize: 480); // 10ms @ 48kHz mono
       
       // Socket.IO: socket.Emit("audio-chunk", opusPacket);
       // WebRTC DC: dataChannel.Send(opusPacket);
   });
   
   recorder.AddModifier(new WebRtcApmModifier(captureDevice, nsEnabled: true, aecEnabled: true));
   recorder.StartRecording();

   Receiver (Remote)

   var queue = new QueueDataProvider(format, maxSamples: 96000, fullBehavior: QueueFullBehavior.Drop);
   var player = new SoundPlayer(engine, format, queue);
   playbackDevice.MasterMixer.AddComponent(player);
   playbackDevice.Start();
   
   // Socket.IO or WebRTC DC message handler:
   socket.On("audio-chunk", (opPacket) => 
   {
       var pcm = opusDecoder.Decode(opPacket);
       queue.AddSamples(pcm); // Thread-safe
   });

   Some tips for production

   1. Opus is non-negotiable for WAN: Cuts raw PCM from ~384KB/s down to ~16-32KB/s while preserving voice clarity. Use Concentus package.
   2. QueueFullBehavior.Drop: Prevents latency spiral when the network stutters. If you notice audible dropouts, bump maxSamples slightly or add a tiny 10-20ms playout delay on the receiver.
   3. WebRTC DC unreliable mode: If you want UDP-like latency with automatic NAT traversal, configure the DataChannel as Reliable: false, Ordered: false. Minor packet loss is handled gracefully by Opus's built-in PLC.
   4. Keep network I/O off the audio thread: QueueDataProvider.AddSamples() is thread-safe, but decode network packets on a background task before pushing to the queue.

   P.S. For the CustomGainComponent, I recommend creating a dedicated modifier instead of relying on a custom SoundComponent, which incurs significant overhead for a simple gain adjustment.
3. LSXPrime

   changed the title [-][QSTN] Your question in a few words[/-] [+][QSTN] Streaming recorded audio with modifiers through Udp[/+] on May 11, 2026
4. atabaysagin commented on May 11, 2026

   

   atabaysagin

   on May 11, 2026

   Author

   More actions

   @LSXPrime Thank you for your reply. I am grateful.
5. atabaysagin commented on May 19, 2026

   

   atabaysagin

   on May 19, 2026

   Author

   More actions

   The echo is still there, I don't know what I did wrong.

   using Concentus;
   using Concentus.Enums;
   using Concentus.Structs;
   using SoundFlow.Abstracts;
   using SoundFlow.Abstracts.Devices;
   using SoundFlow.Backends.MiniAudio;
   using SoundFlow.Components;
   using SoundFlow.Enums;
   using SoundFlow.Extensions.WebRtc.Apm;
   using SoundFlow.Extensions.WebRtc.Apm.Modifiers;
   using SoundFlow.Providers;
   using SoundFlow.Structs;
   using System;
   using System.Buffers;
   using System.Collections;
   using System.Collections.Generic;
   using System.Diagnostics;
   using System.Drawing.Imaging;
   using System.Linq;
   using System.Net.Sockets;
   using System.Text;
   using System.Text.RegularExpressions;
   using System.Threading.Tasks;
   using System.Xml.Linq;
   using static System.Windows.Forms.DataFormats;

   namespace EVKS.App_Code {

   public class IMicrophone {
   
       readonly MiniAudioEngine engine = new MiniAudioEngine();
   
       readonly AudioFormat format = AudioFormat.DvdHq;
   
       FullDuplexDevice fullDuplexDevice;
   
       Recorder? recorder;
       SoundPlayer? player;
       public QueueDataProvider queue;
   
       public Action<float[]>? OnAudioCaptured;
   
       readonly List<float> pcmBuffer = new List<float>();
       public int frameSize = 960;
   
       public IMicrophone() {
           fullDuplexDevice = engine.InitializeFullDuplexDevice(playbackDeviceInfo: engine.PlaybackDevices.FirstOrDefault(d => d.IsDefault), captureDeviceInfo: engine.CaptureDevices.FirstOrDefault(d => d.IsDefault), format: format);
           queue = new QueueDataProvider(format, maxSamples: 4800, fullBehavior: QueueFullBehavior.Drop)!;
       }
   
       public void deviceConnection() {
           fullDuplexDevice.Start();
   
           player = new SoundPlayer(engine, format, queue);
           fullDuplexDevice.PlaybackDevice.MasterMixer.AddComponent(player);
           player.Play();
   
           var apmModifier = new WebRtcApmModifier(fullDuplexDevice,
           aecLatencyMs: 40,
           nsEnabled: true,
           aecEnabled: true,
           aecMobileMode: false,
           nsLevel: NoiseSuppressionLevel.VeryHigh);
   
           recorder = new Recorder(fullDuplexDevice.CaptureDevice, (samples, cap) => {
               lock (pcmBuffer) {
   
                   pcmBuffer.AddRange(samples);
                   while (pcmBuffer.Count >= frameSize) {
                       float[] frame = new float[frameSize];
                       pcmBuffer.CopyTo(0, frame, 0, frameSize);
                       pcmBuffer.RemoveRange(0, frameSize);
                       OnAudioCaptured?.Invoke(frame);
                   }
               }
   
           });
           recorder.AddModifier(apmModifier);
           recorder.StartRecording();
       }
   
       public void deviceStop() {
           recorder?.Dispose();
           player?.Dispose();
           fullDuplexDevice.Dispose();
           queue?.Dispose();
           engine?.Dispose();
       }
   }
   

   }