Error: No room to store incoming packet

[msanis]

Hello

We are experiencing "No room to store incoming packet" error on certain conditions. The source multicast is from a professional encoder (Tandberg)

Condition 1: When TS bit rate of source is up to 16Mbps, then SRT works perfect

Condition 2: When we increase the TS bit rate higher (eg. 17 Mbps), then we see the below error message
RX side:
Accepted SRT source connection
18:11:36.374843/SRT:RcvQ:workerE: SRT.c: %1005060842:No room to store incoming packet: offset=0 avail=0 ack.seq=489148889 pkt.seq=489148889 rcv-remain=8191
18:11:36.375572/SRT:RcvQ:workerE: SRT.c: %1005060842:No room to store incoming packet: offset=1 avail=0 ack.seq=489148889 pkt.seq=489148890 rcv-remain=8191

TX side:
155288 bytes lost, 21560028 bytes sent, 31287900 bytes received
15353772 bytes lost, 21560028 bytes sent, 46486384 bytes received
31324748 bytes lost, 21560028 bytes sent, 62457360 bytes received
47340468 bytes lost, 21560028 bytes sent, 78473080 bytes received
63335132 bytes lost, 21560028 bytes sent, 94467744 bytes received

Condition 3: We start SRT streaming initially with 16Mbps and when it is fully functioning, we increase the source TS bit rate from the encoder settings on the fly to 25Mbps. Still SRT continues to function without any errors and at the RX side we see 25Mbps on output multicast.

It seems SRT is currently handling traffic only upto 16Mbps. Any suggestions ?

Our commands:
TX: stransmit "udp://@226.24.112.4:2000?ttl=64" "srt://1.2.3.4:7005?mode=caller&latency=5000" -t:-1 -s:3000

RX: stransmit "srt://:7005?mode=listener" "udp://@225.10.10.10:2000?ttl=64" -t:-1 -s:3000 -v

[ethouris]

The "No room to store incoming packet" message appears when the RX buffer gets completely full and is waiting for the receiving application to extract the waiting packets. This rcv-remain data shows how many buffers (each one containing data transmitted by a single UDP packet, typically 1316 bytes) are still waiting until the application extracts them.

If you are using the SRT 1.2 version stransmit, just note that this was at that time only a testing tool and choking on the output medium may prevent it from reading from SRT medium on time. Although I have never seen it happening on UDP output, only with file://con output with pipeline to another application.

I would keenly see the debug logs from this session and see how fast the data extraction on RX happens, especially at the moment when the RX buffer level increases.

Mind also that the "latency control" facility (TSBPD) is using the RX buffer to magazine the incoming packets until their "time to play" comes. I can see that you have set quite a long latency (5 seconds) and mind that effectively the RX buffer will have to magazine packets for 5 seconds of stream constantly. Might be that with high bitrates combined with high latency the RX buffer size might be too small, or at least balance on the edge. You can change the size of the receiver buffer through the rcvbuf option (mind that it will be fixed to not exceed the fc option). The default value for rcvbuf is 8192 and for fc - 25600.

[rmsoto]

Hello Ethouris,

I´m also having a similar issue trying to send an hevc 60mbs TS, I can only get 42,5 at output of SRT whily trying the same with a 25TS its fine... where you can change the parameters you said above?

something like this?

./srt-live-transmit -v -r:30 -s:30 -t:0 "udp://@224.168.204.41:2041" "srt://:9001?latency=500&sndbuf=8192&fc=25600"

./srt-live-transmit -v -r:30 -s:30 -t:0 "srt://127.0.0.1:9001?latency=500&rcvbuf=8192&fc=25600" "udp://@224.168.204.250:2250"

Thanks

[jeandube]

@rmsoto you may want to set the max output bw that is capped at 30Mbps by default (this option is not working well and that could explain the 42.5 you get). The option "maxbw" is in bytes/sec.

[jeandube]

Sorry, I just read this issue again, my proposal would not fix the 'no room' on the receiver. but only ensure proper drainage of the sender's buffers.

[rmsoto]

Hi @jeandube , the mxbw helped to achieve the same output bitrate than the input one but I still get the error:

14:44:53.373608/SRT:RcvQ:worker*E: SRT.c: %328810773:No room to store incoming packet: offset=0 avail=0 ack.seq=2053936868 pkt.seq=2053936868 rcv-remain=8191

_increasing the parameters rcvbuf=16384&fc=51200 does not make any good result as I get less bw than expected...I think I´m not introducing the correct units...

Thanks

[jeandube]

@rmsoto Good if you fixed the sender but the noroom problem on the receiver is an indication that the receiving app is not draining the received packets fast enough. Increasing the received buffer just delay the problem. By fixing the sender you just exacerbated the receiver's problem.

[rmsoto]

I think its now working better since we changed the values of sdnbuf, rcvbuf and flight window inside core.cpp and recompile...let’s see how it works during 24h

Thanks!

[haozhenxiao]

@ethouris I also got the same error and I checked the number of packets dropped by the kernel is 0: RX errors 0 dropped 0 overruns 0 frame 0, as you mentioned

The "No room to store incoming packet" message appears when the RX buffer gets completely full

In this case, the kernel should start dropping packets, right? Or I misunderstood RX buffer you mentioned?

[ethouris]

Again: this "No room to store incoming packet" may happen in two cases:

1. The application works slowly and doesn't do the calls to srt_recvmsg even though when it did it would read the packet of data immediately (IOW, it's not the app waiting for the data from SRT, but it's SRT waiting for the application to pick up what is already delivered to it).

2. In some strict cases it may be the problem of too small reception buffer due to TSBPD. TSBPD keeps packets in the buffer until the "time to play" comes, so if you multiply the latency by the bitrate expressed as a number of 1316-byte packets, it should give you the size of the buffer used by TSBPD. By doing it in reverse you'll get the maximum time of the video that can be stored in the receiver buffer (and compare it to latency - the part taken by latency should be much less than the rest of the buffer).

Both these things happen when the packet has already passed the network link, that is, the work for transporting the packet through the network has been already successfully finished.

May happen that the application experiences some temporary slowdown, which will get evened up later; in this case increasing the size will help. But not in case when the application is systematically increasing the delay between the moment when the packet is given up to the application and the moment when the application reads it - in this case a buffer of any size will eventually be overflown.

[haozhenxiao]

@ethouris so RX buffer is not the linux kernel buffer, right? I increased rmem_default and rmem_max but still getting the same error.

[ethouris]

We are talking SRT here, the userspace solution, which uses instrumentally UDP to implement the transmission. It provides its own buffers that work according to the rules it defines. Kernel parameters have nothing to do with this.

Let me be more precise about the above: both these things happen when the packet has already passed the network link and all system buffers and is about to be stored by SRT in its private buffers.

[ethouris]

As you have once faced this problem that is likely to result from performance problems, would you be able to test the version from this branch?

https://github.com/ethouris/srt/tree/dev-rcvworker-sleeps