Incremental delivery of gargantuan amounts of audio data from OfflineAudioContext rendering

[JohnWeisz]

Describe the issue

Currently, OfflineAudioContext.startRendering will unconditionally and always result in the creation of an AudioBuffer containing the rendered audio, regardless of whether it is actually used or not.

This is a problem when used to render a massive duration, as it results in a gargantuan memory impact. Take the following settings for example:

• Sample rate: 48kHz
• Channels: 4
• Duration: 4 hours

Rendering this monster, we have 48000 * 4 * 4 * 60 * 60 = 2_764_800_000 float32 values total, which amounts to more than 11 Gigabytes of in-memory data (more specifically, we don't, because no end-user browser will allow this allocation without flags, some even with flags, not speaking of systems which don't even have this much RAM in the first place).

Where Is It

https://webaudio.github.io/web-audio-api/#dom-offlineaudiocontext-startrendering, more specifically in step 4 of the OfflineAudioContext.startRendering process:

Create a new AudioBuffer, with a number of channels, length and sample rate equal respectively to the numberOfChannels, length and sampleRate values passed to this instance’s constructor in the contextOptions parameter. Assign this buffer to an internal slot [[rendered buffer]] in the OfflineAudioContext.

Additional Information

To avoid the above issue, OfflineAudioContext should "somehow" allow skipping this preliminary forced memory allocation, and be able to deliver data at regular intervals.

The way I see it, there are several options here, and I personally think the MediaRecorder.ondataavailable is a good starting point, which is an already established example of incremental data delivery.

1. Having OfflineAudioContext expose a ReadableStream that's pushed as data is available from rendering
   • Note: this has the potential to synergize extremely well with the proposed WebCodecs API, which has a proposal to work out of a ReadableStream -- if the OfflineAudioContext exposed a ReadableStream, it would be possible to plug it into the WebCodecs API directly and acquire formatted audio files in an efficient and elegant manner, which is a very strong point for this option
2. Emitting some sort of a dataavailable event, possibly with an AudioBuffer for each event containing a chunk of the rendered data
   • Probably unnecessary, as AudioWorklet is a viable alternative already
3. Rendering to a Blob (I personally don't see how the details would work out here)
4. Simply not returning anything, and instead letting user code work it out using AudioWorklets (emitting custom "events", building a stream, etc.)

Polyfilling

While the memory impact is unavoidable with the current implementations of startRendering, for some options above, a polyfill based on an AudioWorkletNode (or even ScriptProcessorNode) could easily make the incremental data delivery an option early on, allowing for prototyping the API in advance and allowing progressive upgrades as the change lands in browsers.

[JohnWeisz]

TL;DR: any chance for an API that does not require allocating an AudioBuffer for the entire output?

[guest271314]

Am not certain

TL;DR: any chance for an API that does not require allocating an AudioBuffer for the entire output?

One option is MediaElementAudioSourceNode https://webaudio.github.io/web-audio-api/#mediaelementaudiosourcenode.

[JohnWeisz]

To be very concise, let me offer a simple example API proposal that is similar to startRendering, but does not result in the creation of an AudioBuffer. The use case is to let user code store the rendering output at their own discretion using an AudioWorklet (which is one of the options I mentioned in the issue):

interface OfflineAudioContext {
    startDestinationlessRendering(): Promise<void>;
}

Naming is purely for letting me demonstrate how this would look, and there are probably way better alternatives for this API (such as a parameter to startRendering, I'm unsure).

Otherwise, this is the exact same method as startRendering, except:

• No AudioBuffer is allocated, and the returned Promise resolves without a value
• Rendered audio samples are not rendered to any destination by default, and user code must manually capture the rendered audio data, e.g. using an AudioWorklet

In the spec, this would look something like the following (based on the current specification of https://webaudio.github.io/web-audio-api/#dom-offlineaudiocontext-startrendering):

startDestinationlessRendering()
Given the current connections and scheduled changes, starts rendering audio.

Let [[rendering started]] be an internal slot of this OfflineAudioContext. Initialize this slot to false.
When startDestinationlessRendering is called, the following steps MUST be performed on the control thread:

1. If the [[rendering started]] slot on the OfflineAudioContext is true, return a rejected promise with InvalidStateError, and abort these steps.
2. Set the [[rendering started]] slot of the OfflineAudioContext to true.
3. Let promise be a new promise.
4. Begin destinationless offline rendering (see below).
5. Return promise.

To begin destinationless offline rendering, the following steps MUST happen on a rendering thread that is created for the occasion.

1. Given the current connections and scheduled changes, start rendering length sample-frames of audio
2. For every render quantum, check and suspend rendering if necessary.
3. If a suspended context is resumed, continue to render audio.
4. Once the rendering is complete, queue a task on the control thread’s event loop to perform the following steps:
   • Resolve the promise created by startDestinationlessRendering().
   • Queue a task to fire an event named complete at this instance, using an instance of OfflineAudioCompletionEvent.

---

Now, the way I see it, this is one option, and probably the simplest to implement. I'm unsure if a configurable output format would be better (it would certainly be more work to implement), i.e.:

interface OfflineAudioContext {
    startRendering(outputMode: "audiobuffer"): Promise<AudioBuffer>;
    startRendering(outputMode: "none"): Promise<void>;
    startRendering(outputMode: "stream"): Promise<ReadableStream>; // or just ReadableStream
}

[guest271314]

Perhaps my initial issue description was misleading, what I really meant is that, if startRendering did not result in the creation of an AudioBuffer, an AudioWorklet could be used to stream chunks of rendered audio data to wherever needed, without the memory impact from the forced single-pass allocation.

Chunks of audio data will accumulate in memory this way, that's true, but unless they are all explicitly collected for in-memory storage by user code, they should be subject to garbage collection at the discretion of the user agent.

Since currently AudioWorkletProcessor reads floating point input data 128 bytes per execution how will memory usage be avoided?

• No AudioBuffer is allocated, and the returned Promise resolves without a value

Can you expand on this? Since a Promise can only be fulfilled once and no promise value is set by startBlindRendering() where is the data processed?

Ideally should we not avoid Float32Arrays altogether? That is, a use a modified version of new MediaElementAudioSourceNode(ac, {mediaElement}) and new MediaStreamAudioDestinationNode(ac) in the form of a "device" (w3c/mediacapture-main#654 (comment)) which can be piped to from a ReadableStream or WritableStream.

[JohnWeisz]

@guest271314

Since currently AudioWorkletProcessor reads floating point input data 128 bytes per execution how will memory usage be avoided?

The way I see it, it depends on the use case and how user code uses the output.

If the user explicitly keeps each sample block for in-memory storage, the same amount of memory allocation will happen by the very end of the rendering process. There is probably nothing we can do about it, and probably nothing we should do about it in the first place.

However, if the user chooses to stream these chunks to, say, a remote server immediately after being available (i.e. in each AudioWorkletProcessor.process call), the incrementally allocated sample blocks are incrementally garbage collected as well, assuming no references are kept to them after they have been streamed to the remote location.

Do you my point here? It is true that the same amount of memory allocation will have to happen over the total duration of a rendering pass, but not at the same time, thus the memory use at one point in time can be kept to a minimum, whereas with the current startRendering system all the memory allocation has to happen up-front, and all allocated memory has to be kept for the entire lifetime of the rendering (and afterwards until the AudioBuffer is disposed).

[guest271314]

@guest271314

Ideally should we not avoid Float32Arrays altogether? That is, a use a modified version of new MediaElementAudioSourceNode(ac, {mediaElement}) and new MediaStreamAudioDestinationNode(ac) in the form of a "device" (w3c/mediacapture-main#654 (comment)) which can be piped to from a ReadableStream or WritableStream.

I see your point now, and I think you are totally right, we should. If we still need Float32Arrays in the end, we can just read from the "device" (or ReadableStream) after all, right?

My original suggestion also included using a ReadableStream as well, although a "device" would be probably even more flexible indeed.

How would this look?

Would look similar to how Chromium, Chrome implement fake audio manager and fake media stream for testing (see linked closed Media Capture and Streams issue) without necessarily storing the file locally, rather streaming the input ReadableStream to an output "device" or File object that can be written to and read using Native File System then the file removed either from the sandboxed directory or actual user filesystem. See this proof-of-concept https://gist.github.com/guest271314/59406ad47a622d19b26f8a8c1e1bdfd5 and this issue puppeteer/puppeteer#5317.

chromium-browser --allow-file-access-from-files --autoplay-policy=no-user-gesture-required --use-fake-device-for-media-stream --use-fake-ui-for-media-stream --use-file-for-fake-audio-capture=$HOME/test.wav%noloop --user-data-dir=$HOME/test 'file:///home/user/testUseFileForFakeAudioCaptureChromium.html'

// use at main thread
navigator.mediaDevices.getUserMedia({audio: true})
.then(mediaStream => {
  const ac = new AudioContext();
  const source = ac.createMediaStreamSource(mediaStream);
  source.connect(ac.destination);
});

The writing the WAV file to local filesystem part can be modified to be a virtual media device capable of being enumerated by getUserMedia() and enumerateDevices() and by any other calling code.

ReadableStream => .pipeTo(new WritableStream() /* a virtual device */) => <MediaStreamTrack|MediaStream (where multiple tracks could be piped through virtual device, audio and video input)>

[guest271314]

@guest271314

That is, a use a modified version of new MediaElementAudioSourceNode(ac, {mediaElement}) and new MediaStreamAudioDestinationNode(ac) in the form of a "device"

Just to clarify, are you proposing that the effective "audio source" behind the stream would be the output from the OfflineAudioContext, as it renders the audio? (i.e. with variable "playback speed" when it comes to the observable audio)

Concretely have proposed and am proposing that the existing media decoders already shipped with browsers be exposed without having to use HTMLMediaElement or decodeAudioData() (w3c/mediacapture-record#166, whatwg/html#2824, whatwg/html#2981, et al.).

Input: ReadableStream, WritableStream
Transform Stream: Existing media decoders, encoders already shipped with browser; WebCodecs; etc.; virtual media (audio, video) device
Output: MediaStream, MediaStreamTrack

[JohnWeisz]

@guest271314

Concretely have proposed and am proposing that the existing media decoders already shipped with browsers be exposed without having to use HTMLMediaElement or decodeAudioData()

I understand this part and I agree with the effort.

Input: ReadableStream, WritableStream
Transform Stream: Existing media decoders, encoders already shipped with browser; WebCodecs; etc.; virtual media (audio, video) device
Output: MediaStream, MediaStreamTrack

Can you elaborate on where OfflineAudioContext would fit here? To me, it seems like returning a ReadableStream from startRendering would solve the issue at hand, and still be flexible enough to support a variety of use-cases, including (1) conversion through the Web Codecs API, (2) any custom data processing using a ReadableStreamDefaultReader if needed, and (3) storage by streaming to a file or a remote destination.

So, I have to admit I'm unsure what the advantages of a virtual device are when it comes to OfflineAudioContext rendering.

[guest271314]

From perspective here would proffer the design of an OfflineAudioContext.<extension_method> output a MediaStreamTrack and avoid AudioBuffer altogether (akin to the WebCodecs proposal and suggestions, referrals thereto). Or, provide that as an option to toggle AudioBuffer for MediaStreamTrack, where applicable.

Given the issue with using AudioBuffer for media input that could be indefinite, without a pre-determined duration or metadata indicating EOF, what is the compelling use case for using AudioBuffer within the algorithm at all, other than to pipe the data through a Web Audio API node to alter the original input to produce a modified audio output?

[guest271314]

An alternative perspective would be to "fix" AudioBuffer to be capable of being "garbage collected" directly by user action or progammtically. Similar to how a File written to user file system using Native File System is removed from local file system when write.close() is not executed, see https://bugs.chromium.org/p/chromium/issues/detail?id=1042018#c6, yet without having to wait for the tab to be closed.

[guest271314]

Either of the above alternatives would mean that the media input would not be retained at all, thus no seeking after a single playback, which is perhaps one rationale for not immediately removing the stored data of an AudioBuffer in memory? MediaSource allows for playback at initial metadata received and following bytes of media in first execution of appendBuffer() and for seeking and multiple playbacks of the media without the memory usage that AudioBuffer does. It would be left to the front-end to store the single use media stream either as an Array with values, ArrayBuffer or by using MediaRecorder, where possible.

[guest271314]

Since for all intents and purposes input can always be a File or Blob and File and Blob have stream() method and Native File System provides a means to write temporary and persisten files we should be able to do

[audioFile, videoFile, otherCodec, textTracks, nthAudioFile, ImageBitmap, readableStream] => <<programmatically created device|WebCodecs|AudioWorklet|OffscreenCanvas|Text> => <writableStream => FileHandle>> => output <MediaStream|MediaStreamTrack> => <OffscreenCanvas_in_Worker|AudioWorklet <use_current_time>> => <MediaStreamDestinationNode<canvasCaptureMediaStreamTrack>>

[rtoy]

See WebAudio/web-audio-api-v2#6. Perhaps that will work for you. Actual API has not yet been determined.

[JohnWeisz]

@rtoy

Thanks, indeed, as per the resolution from that issue, one of the options specified:

Integrate with WHATWG Streams to be able to stream the rendering of PCM data to something else. This would work by outputting 128 frames buffers, and the final one potentially less than 128 frames (depending on the number of frames requested initially).

I personally feel this is the ideal answer, considering the proposed Web Codecs API, as well as the generally advantageous nature of streaming when it comes to audio processing. Is there anything I can do to help advance matters on this end?

The good news is, today it's already possible to use an AudioWorklet to build a ReadableStream, containing the stream of float32 pcm values, which -- in case the API is defined and ready for implementation -- can be used for a rather efficient polyfill straight away.

Note: ironically, I was the author of that issue as well, although with a considerably different intention at the time (render cancellation, while this issue aims to discuss a forced hard limitation in the current implementation; needless to say, it seems there is a single solution to both).

[guest271314]

One observation re

4. Once the rendering is complete, queue a task on the control thread’s event loop to perform the following steps:

• Resolve the promise created by startDestinationlessRendering().

is that if we await a Promise yet a n th startDestinationlessRendering() Promises are queued how will perceptible gaps in playback be overcome (relevant to microtasks)? Even when calling globalThis.console.log() can affect playback. How can sequential queued startDestinationlessRendering() Promises be fufilled and processed in AudioWorkletProcessor without any perceptible gap between fulfilled Promises?