Description
NetEQ jitterbuffer
Video frames can be displayed as fast as they can be received as a result of playout-delay, jitterBufferTarget, and other WebRTC settings.
However, the minimum latency for Opus audio is constrained by NetEQ. This behavior is not well-researched, but when understood, this may decrease the remaining bits of latency the audio stack has.
https://chromium.googlesource.com/external/webrtc/+/master/modules/audio_coding/neteq/g3doc/index.md
https://bloggeek.me/webrtc-discard-media-packets/
Moreover, while I have specified 2.5 (3) ms as the frame size and ptime of the Opus packets, this might not have an actual effect. This behavior should be clearly verified. Edit all sections with OPUS_FRAME: to modify this.
Opus + RED
An issue required to be solved from upstream, tracked in: https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/3550
Described below:
There is a lack of redundancy available with the audio stack, especially with Opus.
This causes concealment events which lead to disconnections in audio in all but the most reliable wired ethernet connections.
The inband-fec parameter of Opus doesn't work with restricted-lowdelay (CELT-only) configurations that most people would want when streaming any other type of media other than live voice (SILK).
An alternative (enabled by default with Chrome 96, other links specifying flags are before Chrome 96) is to use RED with Opus.
https://groups.google.com/g/discuss-webrtc/c/5761etCrSuA/m/hEJto5xwBgAJ
https://webrtc.github.io/samples/src/content/peerconnection/audio/
https://webrtchacks.com/red-improving-audio-quality-with-redundancy/
https://www.meetecho.com/blog/opus-red/
https://webrtchacks.com/implementing-redundant-audio-on-an-sfu/
This is what many providers (including Zoom) seem to use, and there are reports (above) that even in jittery conditions, RED with distance=2 improves concealment from 60% to 18% and 20% to 0.7%.
The current GST_WEBRTC_FEC_TYPE_ULP_RED is meant for video, and along with the string red/48000 instead of red/48000/2 that Opus requires, doesn't work overall. When manually munging the red/48000/2 string into the SDP and using rtpredenc with request-fec-encoder, RED packets are sent correctly, but those packets (2 times larger than the Opus bitrate with distance=2) are instead played by Chrome's WebRTC stack, not Opus, contrary to https://webrtc.github.io/samples/src/content/peerconnection/audio/.
I propose that this capability be called GST_WEBRTC_FEC_TYPE_OPUS_RED.