SDL2: Audio recording, multiple audio devices. (627)

[GalacticEmperor1]

Issue №627 opened by illume at 2018-11-16 08:54:26

This is about exposing:

1. different SDL2 audio devices
2. audio input recording

# 1) Selecting device names

SDL2 can use different audio devices, and can select them based on device name and also if it is an input or output device.

# # multiple device names

I've added a mixer.init(devicename=''), mixer.get_audio_device_name, mixer.get_num_devices. So you can choose the sound device you want to use the mixer with.

Unfortunately, you need to initialize the SDL audio system before those two functions work. And currently that is done in pygame with pygame.mixer.init(). Luckily it's possible to quit the mixer, and initialize it again. But still... sort of annoying.

# 2) Use cases for audio input

For audio input, I guess there's a few use cases:

1. using audio input as a controller in a game (pitch detection, beat detection, loudness, etc)
2. karaoke (audio to be played back too).
3. audio over the network
4. saving audio data to files.
5. music editing program (audacity)

# # proposed API for recording.

# mixer.record
def record(
	devicename:str,
	frequency:int, 
	audioformat:int, 
	numchannels:int, 
	chunksize:int, 
	audiocallback:Callable[bytes, int, int, int, int]):
    """ Starts recording audio in the requested way.

    :param devicename: One of the device names from get_audio_device_name.
                                      If None is passed in, it uses the default recording device.
    :param frequency: Number of samples per second. 44100, 22050, ...
    :param audioformat: AUDIO_F32SYS, AUDIO_F32SYS, AUDIO_U16SYS, AUDIO_S16SYS, ...
    :param numchannels: 2 if stereo, 1 if mono.
    :param audiocallback: called when chunksize amount of samples are available.
    """

def record_pause(devicename=None):
    """ pause the recording.
        :param devicename: One of the device names from get_audio_device_name.
                                          If None is passed in, it uses the default recording device.
    """

# # Example usage of record API.

def audiocallback(samples:bytes, frequency, audio_format, numchannels, chunksize):
    """ called every time there is audio data available.

    'samples' are audio data. Copy into to your own buffer, or process it quickly.
    
    Some things you could do in this callback:
         - put into a queue.Queue, so another thread can process it.
         - post a copy into the pygame event queue.
         - do beat detection, or pitch detection(if it is really fast).
         - copy the data into a pool of sample buffers to avoid memory allocations.
    """
    assert len(samples) == numchannels * chunksize * mixer.samplewidth(audioformat)

mixer.record(
    devicename=None,
    frequency=44100, 
    audio_format=pg.AUDIO_F32SYS,
    numchannels=1,
    chunksize=512,
    audiocallback=audiocallback,
)

mixer.record_pause(devicename=None)

---

Comments

# # illume commented at 2018-11-17 05:19:38

Had second thoughts about this.

Instead, I think perhaps a pygame.audio.AudioDevice() which is closer to SDL2 might be better.

So I'm working on what that would look like.

---

# # robertpfeiffer commented at 2019-09-26 13:11:21

At least one other member of the Discord also wants to record video by piping data from a pygame game into ffmpeg.
One big problem for recording purposes is synchronising audio with video. We can dump the video every frame into files or pipe to ffmpeg. How do we sync video frames to audio, other than maintaining a perfect framerate? If I want to record a GIF, I don't need to keep the framerate up, I can have my game run at 10FPS as long as I dump every frame. I would never drop frames during GIF recording, but as long as the resulting gif isn't missing frames, I can set it to play back at 30FPS and everything is fine.
At least one of my games had a slow mode so I wouldn't mess up during GIF recording. (like this https://twitter.com/blubberquark/status/701816641018003460).
As far as I understand it, the audio recording runs at wall clock time. What I would want would be a function I call once per frame with the elapsed in-game time to get a slice of mixed audio. It doesn't even have to be played on the actual speaker, it could be a thing that works with dummy audio only.