Memory problem with GOP cache enabled

[tuan3w]

Hi @winshining,
When I start nginx, it consumes about a little memory. When i simulate concurrent downloads using wrk tool, memory of my machine starts to increase up to 4.5G. However, after benchmark, I see that the memory doesn't release at all.
Beside that, I have several questions about GOP cache. Is the size of memory for caching proportional to the number of subscriber/players when GOP cache enable ? Is there any way for me to control size of GOP cache?
Thanks.

[winshining]

Did you use the latest commit? Some guys posted the same problem as you mentioned. However, I think I've solved it in codes after release v1.2.3. Here is my benchmark:
500 clients (HTTP-FLV) simulated by srs-bench:
[2018-05-04 19:15:57.103] [report] [1789] threads:500 alive:500 duration:2400 tduration:0 nread:127.67 nwrite:0.00 tasks:500 etasks:0 stasks:0 estasks:0
[2018-05-04 19:16:27.106] [report] [1789] threads:500 alive:500 duration:2430 tduration:0 nread:127.66 nwrite:0.00 tasks:500 etasks:0 stasks:0 estasks:0
[2018-05-04 19:16:57.108] [report] [1789] threads:500 alive:500 duration:2460 tduration:0 nread:127.51 nwrite:0.00 tasks:500 etasks:0 stasks:0 estasks:0
Memory consumed by nginx:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1535 nobody 20 0 112m 52m 1100 S 11.7 0.9 4:00.14 nginx
1534 root 20 0 67800 5608 336 S 0.0 0.1 0:00.00 nginx

Answers for GOP cache:
1 No. There is only one GOP cached by publisher, after receiving a complete GOP, gop_cache module will update the GOP chain.
2 No. You can hack the code and modify it:)

[winshining]

And here is my nginx.conf:

server {
    listen 1935;
    server_name *.xxx.org;

    out_queue   4096;
    out_cork    16;
    max_streams 128;
    wait_video  on;

    chunk_size 4000;

    application myapp {
        live on;
        gop_cache on;
    }
}

[tuan3w]

Hi @winshining,
I'm using latest version. Nginx version is 1.13.12. Here is my setup.
https://gist.github.com/tuan3w/3230988b23f13cd30ce59f7a91321c44

[winshining]

Hi @tuan3w:
Thank you for your reply.
I used config similar to what you provided, and Nginx version I used is 1.13.10.
The following is my benchmark:
./wrk -d 5m -t10 -c500 http://localhost:8080/live?app=myapp\&stream=mystream
Running 5m test @ http://localhost:8080/live?app=myapp&stream=mystream
10 threads and 500 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 0.00us 0.00us 0.00us -nan%
Req/Sec 0.00 0.00 0.00 -nan%
0 requests in 5.00m, 4.55GB read
Requests/sec: 0.00
Transfer/sec: 15.53MB
And the following is statistics about nginx:
top - 00:47:11 up 3:12, 5 users, load average: 0.17, 0.53, 0.75
Tasks: 5 total, 0 running, 5 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.0 us, 3.9 sy, 0.0 ni, 93.8 id, 0.3 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem: 3854532 total, 3366588 used, 487944 free, 104024 buffers
KiB Swap: 2147324 total, 0 used, 2147324 free, 1700528 cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
32420 www-data 20 0 62952 15m 1468 S 0.0 0.4 1:00.21 nginx
32423 www-data 20 0 60884 13m 1348 S 0.3 0.4 0:50.72 nginx
32422 www-data 20 0 60480 13m 1548 S 0.3 0.3 0:49.41 nginx
32421 www-data 20 0 59564 11m 1320 S 0.0 0.3 0:43.96 nginx
32419 root 20 0 49692 1284 236 S 0.0 0.0 0:00.00 nginx
So there is no memory leak in the latest version now.

[tuan3w]

Hi @winshining,
It looks fine for me when I do stress test on localhost. Maybe there isn't enough stress on the system. Can you test again with
http://play-with-docker.com
I created five instances to do stress test. After a hour, the memory is still high.

[winshining]

Sorry to reply you after a long time.
I deployed nginx and srs_http_load on two different hosts and the following is my benchmark:
1000 clients simulated by srs-bench:
[2018-05-06 00:18:12.193] [report] [6015] threads:1000 alive:1000 duration:30 tduration:0 nread:212.90 nwrite:0.04 tasks:1000 etasks:0 stasks:0 estasks:0
[2018-05-06 00:18:42.195] [report] [6015] threads:1000 alive:1000 duration:60 tduration:0 nread:213.22 nwrite:0.02 tasks:1000 etasks:0 stasks:0 estasks:0
[2018-05-06 00:19:12.205] [report] [6015] threads:1000 alive:1000 duration:90 tduration:0 nread:231.73 nwrite:0.01 tasks:1000 etasks:0 stasks:0 estasks:0
[2018-05-06 00:19:42.203] [report] [6015] threads:1000 alive:1000 duration:120 tduration:0 nread:235.94 nwrite:0.01 tasks:1000 etasks:0 stasks:0 estasks:0
[2018-05-06 00:20:03.012][33][error] read header from server failed. ret=104 errno=104(Connection reset by peer)
[2018-05-06 00:20:03.013][33][error] parse response body failed. ret=104 errno=104(Connection reset by peer)
[2018-05-06 00:20:03.013][33][error] http client parse response failed. ret=104 errno=104(Connection reset by peer)
[2018-05-06 00:20:03.013][33][error] http client get url failed. ret=104 errno=104(Connection reset by peer)
[2018-05-06 00:20:03.015][982][error] read header from server failed. ret=104 errno=104(Connection reset by peer)
[2018-05-06 00:20:03.015][982][error] parse response body failed. ret=104 errno=104(Connection reset by peer)
[2018-05-06 00:20:03.015][982][error] http client parse response failed. ret=104 errno=104(Connection reset by peer)
[2018-05-06 00:20:03.015][982][error] http client get url failed. ret=104 errno=104(Connection reset by peer)
[2018-05-06 00:20:12.203] [report] [6015] threads:1000 alive:1000 duration:150 tduration:0 nread:237.27 nwrite:0.01 tasks:1002 etasks:2 stasks:0 estasks:0
[2018-05-06 00:20:42.203] [report] [6015] threads:1000 alive:1000 duration:180 tduration:0 nread:242.77 nwrite:0.01 tasks:1002 etasks:2 stasks:0 estasks:0
[2018-05-06 00:20:43.723][630][error] read header from server failed. ret=104 errno=104(Connection reset by peer)
[2018-05-06 00:20:43.723][630][error] parse response body failed. ret=104 errno=104(Connection reset by peer)
[2018-05-06 00:20:43.723][630][error] http client parse response failed. ret=104 errno=104(Connection reset by peer)
[2018-05-06 00:20:43.723][630][error] http client get url failed. ret=104 errno=104(Connection reset by peer)
[2018-05-06 00:21:12.204] [report] [6015] threads:1000 alive:1000 duration:210 tduration:0 nread:238.88 nwrite:0.01 tasks:1003 etasks:3 stasks:0 estasks:0
[2018-05-06 00:21:42.204] [report] [6015] threads:1000 alive:1000 duration:240 tduration:0 nread:239.54 nwrite:0.00 tasks:1003 etasks:3 stasks:0 estasks:0
[2018-05-06 00:22:12.204] [report] [6015] threads:1000 alive:1000 duration:270 tduration:0 nread:238.02 nwrite:0.00 tasks:1003 etasks:3 stasks:0 estasks:0
[2018-05-06 00:22:35.838][393][error] read header from server failed. ret=104 errno=104(Connection reset by peer)
[2018-05-06 00:22:35.838][393][error] parse response body failed. ret=104 errno=104(Connection reset by peer)
[2018-05-06 00:22:35.838][393][error] http client parse response failed. ret=104 errno=104(Connection reset by peer)
[2018-05-06 00:22:35.838][393][error] http client get url failed. ret=104 errno=104(Connection reset by peer)
[2018-05-06 00:22:42.232] [report] [6015] threads:1000 alive:1000 duration:300 tduration:0 nread:239.91 nwrite:0.00 tasks:1004 etasks:4 stasks:0 estasks:0
Statistics about nginx:
top - 00:23:59 up 3 days, 8:34, 2 users, load average: 0.21, 0.29, 0.26
Tasks: 2 total, 2 running, 0 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.2%us, 1.0%sy, 0.0%ni, 98.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8388608k total, 8388608k used, 0k free, 0k buffers
Swap: 4194304k total, 637056k used, 3557248k free, 2634632k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18929 nobody 20 0 153m 102m 1212 R 19.0 1.2 2:25.99 nginx
18927 nobody 20 0 133m 82m 1228 R 22.3 1.0 2:26.30 nginx
And the following is statistics about nginx after I stopped benchmark:
top - 00:25:25 up 3 days, 8:35, 2 users, load average: 0.39, 0.30, 0.27
Tasks: 2 total, 0 running, 2 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8388608k total, 8388608k used, 0k free, 0k buffers
Swap: 4194304k total, 637056k used, 3557248k free, 2634976k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18929 nobody 20 0 141m 90m 1212 S 0.0 1.1 2:39.70 nginx
18927 nobody 20 0 133m 82m 1228 S 0.0 1.0 2:40.61 nginx

Some memories were recycled in connections pool by nginx, so they were not all freed.
I can't reproduce the problem as you mentioned:(

[tuan3w]

I updated how you can reproduce the output here. https://gist.github.com/tuan3w/3230988b23f13cd30ce59f7a91321c44

I also made a proof video, you can watch here: https://www.youtube.com/watch?v=SUkcasp83w8&feature=youtu.be

After 10 minutes, my machine still reports that 3.02G memory is used.

[winshining]

[winshining]

BTW, I can not watch the video you uploaded, since it refused to be connected.

[tuan3w]

Hi @winshining ,

I updated video pivacy. Can you recheck again? Maybe the video is abandoned from your country :)).
I looked at your image. Is the url only "./wrk -d ... http://..:8000/live" withoout other parameters ?
You should add quote to url to prevent breaking command due to "&" symbol.

img

Thanks

[winshining]

In fact it was with parameters, but there was not enough space for terminal to display them. When I ran it foreground, it showed the complete command.
I can not still watch the video, the browser told me it was time out to connect it (VPN did not help neither).

[tuan3w]

I uploaded it on google-drive: https://drive.google.com/file/d/1SlKsMq_zDzOzquoqtnStPH4Tc4ySKgCz/view
In the meantime, I will create a docker image for testing.

[winshining]

The video was broken, player could only play 26 seconds (both vlc and quicktime).

[tuan3w]

Please check this link. Not sure why old file is broken :)) . https://drive.google.com/file/d/1DZokdi_hp_uiAyBjZPeHOZvIPlu229TU/view?usp=sharing

[winshining]

@tuan3w The video is OK. I will check what caused the high memory consumption.