Could not find Bottle Neck

[Patagonico71]

Hi Guys!, I’m trying to convert 90 UDP streams to SRT for internet transmission, but I can’t identify the bottleneck. There are no packet losses on the network interface, CPU usage stays below 15–20% per core, and IRQs are properly distributed across the cores.

Honestly, I’m not sure what else to check. Could you please give me a hand?

Here are the hardware details:

CPU: Xeon(R) D-2177NT

RAM: 64 GB

Network connection: limited to 3.8 Gbps

Best regards,
Juan Pablo

[cl-ment]

Hi Juan Pablo,
Thank you for reporting this issue! To help us investigate, could you please provide:

• A detailed explanation of what you're experiencing
• A code sample that reproduces the issue (if possible)
• What you expected to happen versus what actually happened

This will really help us understand the problem better.
Best regards,

[Patagonico71]

Hi cl-ment!, I have multiple UDP streams that I need to send via SRT using a small script. The script takes each multicast signal and converts it, starting from a specific base port (50000), into SRT.

On the receiving side, I have a Flussonic server that ingests those SRT streams and encodes them into different bitrates.

The problem I’m facing is that when I have more than 20 streams, they start to arrive with errors — basically, packets are being lost. As I increase the number of simultaneous streams, the packet loss grows accordingly.

The SRT command I’m using for each stream is the following: srt-live-transmit "udp://232.15.0.210:50000?localaddr=172.20.11.11&iface=eno11np1" "srt://0.0.0.0:50000?mode=listener&latency=1200" -v -statsout 100

[cl-ment]

Thank you for your answer!
To help troubleshoot further, I have a few questions and suggestions:
Questions:

• What is the bandwidth of each SRT stream?
• Did you monitor CPU usage per core on the receiving side? Re-encoding can be quite demanding and might be causing issues.

Potential causes and solutions:

• We've found that network nodes like firewalls can sometimes drop packets under high throughput—this might be what's happening in your case.
• Another possibility: Docker containers have network abstraction layers that can also drop packets at high bandwidth. If you're using Docker, this might be worth investigating.
• You could also try increasing your OS UDP buffer sizes on both the sending and receiving sides. For example, on Linux, you can try these commands (or even higher values):

sudo sysctl -w net.core.wmem_max=16777216
sudo sysctl -w net.core.rmem_max=16777216

[ethouris]

There's no iface attribute for UDP medium in srt-live-transmit, nor localaddr. To explicitly set the device in use, use adapter with the IP address assigned to that local device, although this is only useful in case of multicast. Here you have the full description, look for "Medium: UDP".

Also regarding the value you pass to -statsout you probably wanted to pass it to the -stats option, which defines the frequency of stats pickup; the -statsout option defines the filename to write the stats to. I'd recommend also to use the -pf csv to set up the csv format so that you can easily view the statistics in a spreadsheet (I had problems lately with opening a CSV file in Microsoft Excel; LibreOffice Calc works ok.). Once you have the stats, you should be able to see the loss and drop level, as well as the used bitrate per stream.

[Patagonico71]

Hi Cl-ment, I’ve already tried increasing the buffer size and also adjusting the ulimit. The streams vary, some are around 8 Mbps and others up to 20 Mbps. We are not using Docker either. From what you mentioned, I think the issue might be caused by the firewall dropping packets under high load. Thanks, I’ll continue running tests.

Thank you for your answer! To help troubleshoot further, I have a few questions and suggestions: Questions:

• What is the bandwidth of each SRT stream?
• Did you monitor CPU usage per core on the receiving side? Re-encoding can be quite demanding and might be causing issues.

Potential causes and solutions:

• We've found that network nodes like firewalls can sometimes drop packets under high throughput—this might be what's happening in your case.
• Another possibility: Docker containers have network abstraction layers that can also drop packets at high bandwidth. If you're using Docker, this might be worth investigating.
• You could also try increasing your OS UDP buffer sizes on both the sending and receiving sides. For example, on Linux, you can try these commands (or even higher values):

sudo sysctl -w net.core.wmem_max=16777216
sudo sysctl -w net.core.rmem_max=16777216

[Patagonico71]

Yes, my mistake with the iface attribute. I will test with -stats and analyze using multiple streams, since with one stream there are no obvious issues."

There's no iface attribute for UDP medium in srt-live-transmit, nor localaddr. To explicitly set the device in use, use adapter with the IP address assigned to that local device, although this is only useful in case of multicast. Here you have the full description, look for "Medium: UDP".

Also regarding the value you pass to -statsout you probably wanted to pass it to the -stats option, which defines the frequency of stats pickup; the -statsout option defines the filename to write the stats to. I'd recommend also to use the -pf csv to set up the csv format so that you can easily view the statistics in a spreadsheet (I had problems lately with opening a CSV file in Microsoft Excel; LibreOffice Calc works ok.). Once you have the stats, you should be able to see the loss and drop level, as well as the used bitrate per stream.