NvCodec H.264 encoder never applies SetRates() — streams stay at the initial ~300kbps unless re-initialized

[shuuki4]

We run server-side avatar publishers (Python SDK 1.1.5 on linux x86_64, L40S, driver 570.195.03) and spent a couple of days chasing why every H.264 track came out blurry despite max_bitrate=2_500_000. It turned out to be the hardware encode path: when libwebrtc picks the NvCodec encoder (implementation_name=NvCodec in the codec stats), the encoder keeps encoding at whatever bitrate it was initialized with — i.e. the BWE start estimate, ~300kbps — for the entire session. SetRates() updates never reach NVENC.

Easiest to see with a camera source (the default path). 540x960@25 publish, max_bitrate 2.5Mbps — outbound-rtp sampled every 2s:

t= 4.0s  send= 292kbps  target=1722kbps  res=268x480   qp=23.7
t=10.0s  send= 283kbps  target=1847kbps  res=268x480   qp=25.9
t=20.0s  send= 299kbps  target=1848kbps  res=268x480   qp=23.3
t=30.1s  send= 282kbps  target=1848kbps  res=268x480   qp=25.1
t=38.1s  send= 288kbps  target=1848kbps  res=268x480   qp=24.7
t=40.1s  send=1725kbps  target=1486kbps  res=360x640   <- libwebrtc upscales on its own (re-init)
t=42.1s  send=1883kbps  target=1890kbps  res=360x640
t=44.1s  send=2390kbps  target=1948kbps  res=540x960   <- and again

To be clear, we didn't touch anything during this run — the resolution steps are libwebrtc's own quality adaptation reacting to the ramping BWE. The send rate sits at ~290kbps for 36 seconds while the target is ~1.8Mbps, then jumps exactly at those two adaptation steps, which happen to be the only two moments the encoder gets re-initialized. We confirmed the correlation with key_frames_encoded: bitrate only ever changes across a re-init IDR, never between them.

With a screenshare source it's worse, because resolution adaptation is disabled so nothing ever re-inits the encoder — the stream stays at ~290kbps for the entire session while the stats report target_bitrate=2500000 and quality_limitation=none throughout:

t= 4.0s  send= 291kbps  target=2500kbps  qp=29.8  quality_limitation=none
t=58.1s  send= 288kbps  target=2500kbps  qp=30.1  (unchanged for 60s)

As a sanity check, forcing one re-init manually (publishing a single frame at a slightly different resolution) takes the same screenshare stream from 290kbps to 2.5Mbps instantly and permanently.

The cause is visible in webrtc-sys/src/nvidia/h264_encoder_impl.cpp:

• InitEncode() sets rcParams.averageBitRate once, from codec_.startBitrate (~line 176 and 225)
• SetRates() (~line 418) only updates the local configuration_ / codec_ fields and returns. It never calls NvEncoder::Reconfigure() — which exists in the vendored NvEncoder.cpp but has zero call sites in the encoder impl.

So every consumer of the NVENC H.264 path is effectively publishing at the BWE start bitrate unless something happens to re-init the encoder.

The fix looks straightforward: build NV_ENC_RECONFIGURE_PARAMS from the current init params with the updated averageBitRate (and vbvBufferSize) and call encoder_->Reconfigure() from SetRates(). nvEncReconfigureEncoder supports bitrate changes without forcing an IDR, so it should be safe at SetRates cadence, or rate-limited if you prefer. This would mirror what the OpenH264 impl does with SetOption. Happy to put up a PR if that approach sounds right — though you can probably build/validate the NVENC path faster than we can.

Env: livekit (python) 1.1.5 / webrtc-sys 0.3.36 (WEBRTC_TAG=webrtc-51ef663, libwebrtc m137), NVIDIA L40S, driver 570.195.03, Ubuntu 22.04 x86_64.