Enhanced mode: a bufferbloated leg ends up carrying most of the stream while the clean leg sits nearly idle #25

Before you file

• I searched existing issues (including closed ones) and this is not already reported.
• I am on the latest release and the problem still happens there.
• I checked the README's Troubleshooting section.
• Every uplink can reach the receiver on its own, and source routing is set up so each IP in the IP list leaves through its own interface (see Assumptions and Prerequisites).
• If only some links carry traffic, I tried --mode classic to find out whether the enhanced scheduler's quality scoring is the cause.

What happened

Three uplinks: one clean (4.2 Mbit/s, 150 ms queue) and two weak with deep queues (1.5 Mbit/s with an 800 ms queue, 2.0 Mbit/s with a 600 ms queue). The SRT feed is a constant 3.8 Mbit/s.

Expected: most of the traffic on the clean 4.2 Mbit/s leg, little on the two weak legs.

Got (run 3 of 4, default settings): the clean leg was sent 0.48 Mbit/s, the two weak legs were sent 2.66 and 3.84 Mbit/s. That is more than either weak leg can carry (1.5 and 2.0 Mbit/s), so their queues sat at 800 and 600 ms, and total sent was 1.85x the feed (6.99 Mbit/s for a 3.79 Mbit/s feed). The receiver saw heavy loss and resends: 15,651 packets reported lost, 4,116 retransmitted, 3,404 unrecovered drops, 1,766 stream discontinuities.

Same profile, same settings, other runs: run 1 was fine (1.06x, 0 unrecovered), run 2 was bad (1.71x, 1,012 unrecovered drops), run 4 never got a stream through the SRT server (separate, unexplained). So it is intermittent; I do not know what decides it.

I see the same thing on real hardware (three USB LTE modems on a Raspberry Pi 3B+): at one point the worst leg had an ~800 ms standing queue and 2,871 packets in flight and carried the whole stream, while two healthy legs (60-100 ms RTT) were both stall-gated and idle.

My reading of the code (not proven): the in-flight cap in selection/enhanced.rs is cc_target_bps * rtt_min * 1.5 using the link's get_rtt_min_ms(). If a leg builds a standing queue that lasts longer than the RTT windows, its "minimum" RTT becomes the queue delay, so its allowed in-flight grows roughly in proportion (800 ms vs 30 ms is ~25x), while a clean leg's cap stays at 1-2 packets and it is "over cap" more easily. In the lab, bad runs had a weak leg rtt_min of 98-538 ms and good runs 19-28 ms.

Steps to reproduce

Lab on one Linux host, network namespaces with netem/tbf per leg (profile attached as lab_profile_r6_lockin_short.json):

Three veth legs shaped as above: leg 1 4200 kbit/s, 150 ms queue, 8 ms delay, 0.3% loss; leg 2 1500 kbit/s, 800 ms queue, 35 ms delay; leg 3 2000 kbit/s, 600 ms queue, 30 ms delay. No events (static).
Start srtla_send in the namespace (command below), feed it a constant 3.79 Mbit/s MPEG-TS via ffmpeg -re -stream_loop -1 -c copy -f mpegts srt://127.0.0.1:6000?...&latency=8000000.
Run 45 s, measure bytes sent per leg and what the receiver gets. Result over 4 runs of this exact setup: 1 good, 2 bad (1.71x and 1.85x), 1 where no stream appeared. With --no-stall-deselect it was better (9 of 12 clean in earlier runs); with --mode classic it was also bad (1 of 6 clean). I measure the window from 15 s to 45 s (after a 15 s warm-up).

Debug log

srtla_send 4.1.0 (3299825), RUST_LOG=debug, lab run 3 (bad run). Excerpt of the 87,603-line log; full log attached as srtla_send_debug_run3.log.gz

2026-10-14T20:00:23Z  INFO unix control socket listening socket_path="/run/lab-ctl.sock"
2026-10-14T20:00:23Z  INFO starting srtla_send: local_srt_port=6000, receiver=10.200.0.1:5000, ips_file=ips
2026-10-14T20:00:23Z  INFO prometheus /metrics endpoint listening local=Some(127.0.0.1:9099)
2026-10-14T20:00:23Z  INFO listening for SRT on [::]:6000
2026-10-14T20:00:23Z DEBUG uplink IPs loaded: 10.200.1.2, 10.200.2.2, 10.200.3.2
2026-10-14T20:00:23Z  INFO added uplink 10.200.0.1:5000 via 10.200.1.2
2026-10-14T20:00:23Z  INFO added uplink 10.200.0.1:5000 via 10.200.2.2
2026-10-14T20:00:23Z  INFO added uplink 10.200.0.1:5000 via 10.200.3.2
2026-10-14T20:00:23Z  INFO Starting RTT probing for 3 connections
2026-10-14T20:00:23Z  INFO Waiting for probe responses from 3 connections
2026-10-14T20:00:23Z DEBUG No REG1 target selected; awaiting REG_NGP
2026-10-14T20:00:23Z DEBUG REG_NGP from uplink #0
2026-10-14T20:00:23Z  INFO Probe response from connection #0 (RTT: 33ms)
2026-10-14T20:00:23Z DEBUG REG_NGP from uplink #2

[... lines omitted ...]

2026-10-14T20:00:46Z DEBUG 10.200.0.1:5000 via 10.200.2.2 quality recovering: 0.98 (burst: 2080)
2026-10-14T20:00:46Z DEBUG 10.200.0.1:5000 via 10.200.3.2 quality recovering: 0.98 (burst: 540)
2026-10-14T20:00:46Z DEBUG Batch drain: 16 packets ready to send
2026-10-14T20:00:46Z DEBUG 🎯 SRTLA ACK received from 10.200.0.1:5000 via 10.200.3.2: 10 sequences
2026-10-14T20:00:46Z DEBUG 10.200.0.1:5000 via 10.200.3.2: ACK increased window 1000 → 1029 (in_flight=215, fast_mode=true) [ENHANCED]
2026-10-14T20:00:46Z DEBUG 10.200.0.1:5000 via 10.200.3.2: ACK increased window 1030 → 1059 (in_flight=214, fast_mode=true) [ENHANCED]
2026-10-14T20:00:46Z DEBUG 10.200.0.1:5000 via 10.200.3.2: ACK increased window 1060 → 1089 (in_flight=213, fast_mode=true) [ENHANCED]

Counts over the whole run: 71,325 DEBUG, 16,245 WARN, 33 INFO. The WARN lines are almost all "NAK reduced window 1000": 7,492 on via 10.200.2.2 (weak leg, 800 ms queue), 5,089 on via 10.200.3.2 (weak leg, 600 ms queue), 1,347 on via 10.200.1.2 (the clean leg).

srtla_send version

4.1.0 (3299825) [srtla_send]

How you installed it

.deb from a GitHub release

Device and OS

Lab: x86_64 server, Ubuntu 24.04.1 LTS, network namespaces (veth, netem, tbf). Real units: Raspberry Pi 3 Model B+ Rev 1.4, Debian 13 (trixie), arm64, three USB LTE modems.

Stream setup

srtla_send --metrics-bind 127.0.0.1:9099 --control-socket /run/lab-ctl.sock 6000 10.200.0.1 5000 ips
(uplinks 10.200.1.2, 10.200.2.2, 10.200.3.2; no other flags, so enhanced mode with the stall guard on; RUST_LOG=debug.) Uplinks: three virtual links as above (real units: 4G/LTE USB modems, one SIM per carrier). Receiver: irlserver srtla_rec plus irl-srt-server (SRTLA-patched SRT port), SRT latency 8000 ms. Encoder / SRT sender: ffmpeg (libsrt) copying a constant ~3.79 Mbit/s MPEG-TS clip into srt://127.0.0.1:6000.

srtla_send_debug_run3.log
lab_profile_r6_lockin_short.json
srtla_send_debug_run3.log.gz

Link stats

Per-second per-leg figures for run 3 are in the attached run3_per_second_leg_stats.jsonl (bytes sent, queue depth, the sender's own metrics). I did not capture get_stats.

Anything else

--no-stall-deselect helps a lot; an SRT maxbw cap on the ffmpeg side (new + resent packets together) made 12 of 12 runs clean. Those are workarounds; the sender still sends heavily to a leg that is already queueing.
Not tried yet: set_quality {enabled: false} on its own, or unplugging the best leg during a live stream.
Happy to run other impairment profiles; the lab harness takes a JSON profile.

upstream_issue_form_fields.md