Silent Audio Failure When Receiving Mono Audio in Unity SDK

[FushimiYuki]

Problem

When a Unity client receives mono (1 channel) audio from remote participants, the audio silently fails to play with no errors or warnings. The issue is hard to debug because:

• ✅ AudioStream creates successfully
• ✅ AudioSource.isPlaying = true
• ✅ All logs show "success"
• ❌ But no audio is heard

Reproduction Steps

1. Remote participant publishes audio using a mono microphone (most built-in laptop mics are mono)
2. Unity client subscribes to the audio track:

   var audioSource = gameObject.AddComponent<AudioSource>();
   var audioStream = new AudioStream(remoteAudioTrack, audioSource);

3. Result: Complete silence, no error messages
4. Other clients (Web, mobile) hear the audio fine

Environment

• Unity 2022.3+ with default audio settings (stereo output)
• Remote participant with mono microphone
• LiveKit Unity SDK (latest)

Root Cause

Location: client-sdk-rust~/webrtc-sys/src/audio_resampler.cpp

WebRTC's RemixAndResample() returns 0 bytes when converting mono→stereo with same sample rate:

Input:  1 channel, 480 samples, 48kHz
Output: 2 channels, 0 samples, 48kHz  ← samples_per_channel is 0
Result: Empty data, no audio

The function correctly sets num_channels=2 but fails to set samples_per_channel, resulting in empty output.

Impact

• Severity: High - Complete audio loss
• Silent failure: No errors in logs, very hard to debug
• Common scenario: Many users have mono microphones
• All platforms affected: macOS, Windows, Linux, iOS, Android

Workaround

Add this to Runtime/Scripts/AudioStream.cs in OnAudioStreamEvent():

var uFrame = _resampler.RemixAndResample(frame, _numChannels, _sampleRate);

// Workaround for mono→stereo bug
if ((uFrame == null || uFrame.Length == 0) && 
    frame.NumChannels == 1 && _numChannels == 2 && 
    frame.SampleRate == _sampleRate)
{
    // Manual mono to stereo conversion
    int samplesPerChannel = (int)frame.SamplesPerChannel;
    short[] monoData = new short[samplesPerChannel];
    short[] stereoData = new short[samplesPerChannel * 2];
    
    var monoSpan = new Span<byte>(frame.Data.ToPointer(), frame.Length);
    MemoryMarshal.Cast<byte, short>(monoSpan).CopyTo(monoData);
    
    for (int i = 0; i < samplesPerChannel; i++)
    {
        stereoData[i * 2] = monoData[i];     // Left
        stereoData[i * 2 + 1] = monoData[i]; // Right
    }
    
    var stereoBytes = MemoryMarshal.Cast<short, byte>(stereoData.AsSpan());
    _buffer?.Write(stereoBytes);
    return; // Skip normal path
}

// Normal path...
if (uFrame != null && uFrame.Length > 0)
{
    var data = new Span<byte>(uFrame.Data.ToPointer(), uFrame.Length);
    _buffer?.Write(data);
}

This workaround successfully restores audio playback.

Why This Matters

This is a critical usability bug because:

1. Users expect audio to "just work"
2. No error messages make debugging extremely difficult
3. Affects real-world usage (mono mics are common)
4. Works fine in Web SDK, so users won't expect Unity to fail

Note

This workaround is a patch that allows my project to run properly for now. However, I'm unsure if there's a more elegant solution. Would appreciate guidance on whether:

1. This is the recommended approach, or
2. There's a better way to handle this in the SDK

Thanks for looking into this!

[sarwar-hsn]

Thank you

[ladvoc]

@FushimiYuki, thank you for reporting this issue—we will investigate. As for logging, that will be added soon by #167.