Audio mix recorder

[hiroshihorie]

Easily mix and record multiple audio sources (AudioRenderer protocol).
Can add additional source while recording.

TODO:

• Customize recording format
• Optimize code

Example usage

let audioSettings: [String: Any] = [
    AVFormatIDKey: kAudioFormatMPEG4AAC,
    AVSampleRateKey: 48_000,
    AVNumberOfChannelsKey: 1,
    AVLinearPCMBitDepthKey: 32,
    AVLinearPCMIsFloatKey: true,
    AVLinearPCMIsNonInterleaved: false,
    AVLinearPCMIsBigEndianKey: false,
]

let recordFilePath = URL(fileURLWithPath: NSTemporaryDirectory()).appendingPathComponent(UUID().uuidString + ".aac")
print("Recording to \(recordFilePath)...")

let recorder = try AudioMixRecorder(filePath: recordFilePath, audioSettings: audioSettings)
try recorder.start()

// Record local audio
AudioManager.shared.add(localAudioRenderer: recorder.addSource())
// And all remote audio
AudioManager.shared.add(remoteAudioRenderer: recorder.addSource())
// Or individual remote tracks
// RemoteAudioTrack.add(audioRenderer:)

[ilo-nanpa]

it seems like you haven't added any nanpa changeset files to this PR.

if this pull request includes changes to code, make sure to add a changeset, by writing a file to .nanpa/<unique-name>.kdl:

minor type="added" "Introduce frobnication algorithm"

refer to the manpage for more information.

[bcherry]

this is awesome and it works!

I found one bug, not sure what the cause is, but if I do (pseudo-code)

room.connect()
audioRecorder = audiomixrecorder(filePath:"../..")
audiomanager.shared.add(localRenderer: audioRecorder.newSource())
audiomanager.shared.add(remoteRendered: audioRecorder.newSource())

then at disconnect

room.disconnect()
audioRecorder.stop()
audioRecorder = nil

then if I try to connect again, i get a crash. I can fix the crash by manually removing the old sources from the audiomanager delegates. so it suggests that the sources (which should be stored weakly in audiomanager) aren't getting deallocated and maybe there is a retain cycle inside AudioMixRecorder

otherwise this is perfect

[hiroshihorie]

Thanks for testing, will take a look 🙂

[bcherry]

@hiroshihorie update: I missed the "removeAllSources" method. when i call that too at disconnect then the crash no longer happens when the next session starts but instead when it ends. which doesn't really make sense...

[hiroshihorie]

@bcherry Can you try to reproduce the crash again ?

[ladvoc]

I was unable to reproduce the crash following the steps Ben described.

[ladvoc]

This might be out of the scope of this PR, but maybe we could define an AudioSettings struct and related types to modernize the API a bit:

var settings = AudioSettings()
settings.formatID = .mpeg4aac
settings.sampleRate = 48_000
settings.channelCount = 1
settings.pcm.bitDepth = 32
settings.pcm.isFloat = true
settings.pcm.isInterleaved = false
settings.pcm.isBigEndian = false

An additional overload of AudioMixRecorder’s initializer will accept this structure instead of a dictionary, but we would keep the existing dictionary-based overload for Objective-C compatibility.

[hiroshihorie]

@ladvoc I agree we can improve the AudioSettings but maybe we can add that later. I think it's a bit hard to properly cover all settings.