Sever is crashing due to memory swapping

[josecelano]

The droplet has started crashing, probably after the last update.

Just after rebooting:

After a while:

It looks like the Index is using a lot of CPU.

I don't know the reason yet. I think this recent PR could cause it:

torrust/torrust-index#530

Maybe the execution interval (100 milliseconds) is too short for this machine.

cc @da2ce7

[josecelano]

In my PC, with the current interval (100 milliseconds):

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
1342564 josecel+  20   0 2355216  38912  29440 S  20.2   0.1   1:26.22 torrust-index

With 1000 milliseconds:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
1346972 josecel+  20   0 2355216  38656  29696 S   2.3   0.1   0:03.38 torrust-index

With 2000 milliseconds:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
1348942 josecel+  20   0 2353168  37888  29952 S   1.3   0.1   0:00.59 torrust-index

With 3000 milliseconds:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
1350156 josecel+  20   0 2353288  37632  29696 S   1.0   0.1   0:00.61 torrust-index

With 4000 milliseconds:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
1351449 josecel+  20   0 2355216  38400  29440 S   0.7   0.1   0:01.23 torrust-index

With 5000 milliseconds:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
1353683 josecel+  20   0 2355224  37632  29440 S   1.0   0.1   0:02.17 torrust-index

Interval (secs)	Number of torrents imported per hour
1 sec	50 * 3600 = 180000
2 sec	50 * (3600/2) = 90000
3 sec	50 * (3600/3) = 60000
4 sec	50 * (3600/4) = 45000
5 sec	50 * (3600/5) = 36000

[josecelano]

I've deployed a new version with the increased interval 2 hours ago and it hasn't crashed:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   3721 torrust   20   0  750016 499496      0 S  20.5  50.9  23:08.13 torrust-tracker
   3867 torrust   20   0  577576  47760   6164 S   6.3   4.9   6:18.35 torrust-index

[josecelano]

The problem is not solved yet :-/:

[josecelano]

It looks like the problem is memory consumption. The memory consumption increases until the server starts swapping. I think the problem could be:

1. The new tracker statistics importer (even with the new 2-second interval).
2. The new torrent repository implementation with the SkipMap. I don't think so because, in the previous implementation with BTreeMap, we were not limiting the memory consumption either.
3. The tracker is receiving too many requests, so we need to limit the number of requests or the memory consumption.

The good thing is that the service restarts with the docker healthcheck after 2 hours. Looking at the docker containers, only the tracker was restarted, so I guess the problem is with the tracker (option 2 or 3):

docker ps
CONTAINER ID   IMAGE                       COMMAND                  CREATED        STATUS                    PORTS                                                                                                                                       NAMES
ee67d0951541   nginx:mainline-alpine       "/docker-entrypoint.…"   14 hours ago   Up 14 hours               0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp                                                                    proxy
31c65c66be26   torrust/index-gui:develop   "/usr/local/bin/entr…"   14 hours ago   Up 14 hours (healthy)     0.0.0.0:3000->3000/tcp, :::3000->3000/tcp                                                                                                   index-gui
a8b2e8977c5c   torrust/index:develop       "/usr/local/bin/entr…"   14 hours ago   Up 14 hours (healthy)     0.0.0.0:3001->3001/tcp, :::3001->3001/tcp                                                                                                   index
0cc1eefec6b6   torrust/tracker:develop     "/usr/local/bin/entr…"   14 hours ago   Up 10 minutes (healthy)   0.0.0.0:1212->1212/tcp, :::1212->1212/tcp, 0.0.0.0:7070->7070/tcp, :::7070->7070/tcp, 1313/tcp, 0.0.0.0:6969->6969/udp, :::6969->6969/udp   tracker

I'm going to try first switching back to the previous repository implementation with the BTreeMap (there could be some deadlock with the SkiMap). If we still have the problem, we need to limit requests and/or memory consumption.

On the other hand, I think we should improve the container healthchecks. We should not wait for 2 hours to restart the container.

[josecelano]

I've opened another issue to discover why there are so many zombie processes. I think there is an easy way to know if the healthcheks cause the crash on the server or not. I can run a private server with the same resources without requests.

Another thing I could check is the number of requests the tracker is handling. Maybe the number of requests per second has increased a lot, and the problem is the server can't handle those requests. In that case, the only solution would be:

1. Limit the amount of requests (reject requests when the server is busy).
2. Limit the memory consumption (we try to handle requests but remove torrents/peers when the torrent repository is full).
3. Increase the server when CPU or memory usage goes over 85%.
4. All of the previous ones.

[josecelano]

5 minutes after restarting the tracker container, I checked the stats:

udp4_connections_handled: 44661,
udp4_announces_handled: 25175,
udp4_scrapes_handled: 652,

The tracker is handling 234.96 req/sec. I think, in this case, the server could be simply busy, and we need to implement one of the solutions above (1 to 3).

cc @da2ce7

UDPATE:

Those are only the tracker client requests. We also have tracker API, Index, stats importer, ...

[josecelano]

It looks like the period without restarting is longer during the night. Maybe it's because there are fewer tracker requests.

Now there are 350 req/sec

In 2 hours
udp4_connections_handled: 824422,
udp4_announces_handled: 1646227,
udp4_scrapes_handled: 55240,

2525889 -> 350.817916667 req/sec