Allow user-selectable render quantum size

[rtoy]

Describe the feature
Allow an AudioContext and OfflineAudioContext to have a user-selectable render quantum size instead of the current fixed size of 128.

This allows better integration with AudioDeviceClient which can render in many different sizes.

Is there a prototype?
No. Can't prototype this.

Describe the feature in more detail
Basically add a new dictionary member to specify a render size for the constructors. The default is 128, of course.

[rtoy]

Updated AudioContextOptions:

dictionary AudioContextOptions {
  (AudioContextLatencyCategory or double) latencyHint = "interactive";
  float sampleRate;
  unsigned long renderSizeHint = 128;
};

It's a hint to allow the browser to choose something else appropriate to the browser such as a power of two.

Updated BaseAudioContext:

interface BaseAudioContext : ... {
...
  readonly attribute unsigned long renderQuantumSize;
}

renderQuantumSize is the actual size. And named this because the spec uses render quantum everywhere, and calls it render quantum size

[rtoy]

What do do about the 3-arg OfflineAudioContext constructor? Should we allow a fourth arg to specify the render size? Or just say that if you want the new goodness you need to use the constructor with dictionary?

[rtoy]

Closely related to WebAudio/web-audio-cg#9

[rtoy]

Another thing to consider with this issue is what happens with ScriptProcessorNode and its buffer size that is required to be a power of two. Yes, that node is deprecated, but not yet removed so we need to think about this a least.

[jas-ableton]

@rtoy From the uninformed perspective of a front-end developer, I'd be more than happy to just use the dictionary constructor.

Also marking this as related to: https://bugs.chromium.org/p/chromium/issues/detail?id=924426, where it seems we've identified that awkward buffer sizes lead to significant performance hits.

[meshula]

The AudioBus/Channel classes in the implementation don't really like changing from frame to frame; the chromium issue describes buffers that are either 128 or 192 bytes; the glitching is probably changes in buffer sizes being propagated through the graph, possibly a frame late. I think an easier and more reasonable fix is to keep the 128 quantum size, but run the streams through a ring buffer so that the engine is always fed 128 byte chunks to process, and possibly running a chunk ahead of the callbacks. I worked through supporting a variable quantum in LabSound (which was originally a fork of the webkit webaudio sources), and it seems like way more trouble than its worth, especially in the face of alternative simple solutions like a ring buffer. (e.g. https://github.com/dr-soft/miniaudio/blob/master/examples/fixed_size_callback.c) By way more trouble, I mean that most of the convolution filters, such as delays, HRTF, and reverb, all have a dependency on working on power of two chunks, and mismatching those versus the render callback size is bothersome, and without careful thought can introduce new pops and latency. Although I did the work on the variable quantum I am going to abandon that branch...

[rtoy]

I believe all implementations do some kind of FIFO/ring buffer to manage the difference between WebAudio's 128-frame chunks and the underlying HW block sizes.

I have made some changes to Chrome to support this, and even in the simple cases if the block size is changed to some other value, many of the current WPT tests fail because the generated numbers are different from the expected. I do not if that's because I messed up or because that's how things work, or because extra (or less?) round-off happens.

And, as you say, getting anything that uses FFTs as the underlying implementation is a ton of work, and impacts performance. For small sizes, this probably hurts performance quite a bit. For large sizes, this probably helps because we can use larger FFT sizes.

In all, this is a ton of work to get things all working and performing well.

[padenot]

What @meshula describes is what is being done today in implementations. It's however very inefficient in cases where the system buffer size is not a power of two (very common on anything but macOS, if we're talking about consumer setups), so we're changing it. There is no other way to fix this properly: the fundamental issue is that the load per callback cannot be stable if the rendering quantum size is not a divisor of the system callback size, and this means that theoretical maximum load is reduced.

Let's consider a practical example (my phone), that has a buffer size of 192 frames. With a rendering quantum of 128 frames, a native sample-rate of 48kHz, and a system buffer size of 192 frames, the rendering with a ring buffer to adapt the buffer sizes go like this:

iteration #	number of frames to render	number of buffers to render	leftover frames
0	192	2	64
1	192	1	0
2	192	2	64
3	192	1	0
4	192	2	64
5	192	1	0

Because this is real-time, we have 192 * 1000 / 48000 = 4ms for each callback, but sometimes we need to render 256 frames in 4ms, and sometimes 128 frames in 4ms. The total maximum load is therefore 50% of what the phone can do.

The render quantum size is not going to be variable. It's going to be user-selectable at construction, and will be based on the characteristics of the underlying audio stack.

This means we'll have to fix the FFT code indeed, the simple fix being to introduce the ring buffer only there.

[jas-ableton]

Thanks for laying this out so clearly and succinctly @padenot!

The point about various nodes needing to operate on certain buffer sizes is a good one, but as @padenot points out, the common practice (in audio plugin development, for example) is to satisfy these constraints within the implementation of the node itself. If a node/plugin needs to operate on power of two buffers because it does FFTs, it's usually the case that the node/plugin is responsible for doing the necessary buffering internally. Similarly, if a node/plugin needs to do its processing at a certain sample rate, the node/plugin will do the required resampling internally, as opposed to forcing all other nodes in the graph to run at the same sample rate.

[meshula]

The part that made my brain hurt, in the WebKit implementation, was the optimization whereby a bus's channels' buffers can be passed up the chain in the case where a node wouldn't actually affect the buffer. Of course the right answer is that the buffer can't be shared when it hits a node that has a different internal need, and that life is much easier if the render quantum is fixed at initialization, rather than runtime-variable.

[padenot]

This issue is not about dynamically changing the buffer size during an AudioContext life time. It's about deciding on a buffer size at initialization, based on what the system can do.

The buffer sharing technique that you see in WebKit is also present in Gecko, and it's just an optimization.

[padenot]

Virtual F2F:

• Really important for advanced use-cases
• Users want to have this for two reasons
  1. Having bigger buffer size in the process method
  2. Align buffer size with the IO vector size to avoid the issue talked about earlier

[padenot]

Exact shape of the API is TBD. There will have to be provisions to haveScriptProcessorNodestill work. Having a FIFO before it and still send buffers with power of two sizes seem like the way to go.

[rtoy]

From today's teleconf:

We want to allow selecting the desired size, but we also need a way to specify that we want to use whatever the HW-recommended size is.

We probably don't want to allow any possible size, but not sure about the constraints except that requiring a power of two isn't going to work for many Android devices. Perhaps small multiples of powers of two? Maybe something like 2^p*3^q*5^r?