Dispatcher service stops updating poller-stats after losing Redis-connection

[ottorei]

We encountered an issue with dispatcher service on a 4 poller distributed setup where the Redis-server was offline for a few hours. This cause 2 / 4 of the pollers to completely fail. For some reason, a few of them did reconnect but the others did not. Maybe the other poller connected at slightly different time when the Redis came back.

The issue can be fixed by manually restarting the service with systemctl. These are the last lines on log.

[maesbrisa]

Hi!

We found the same issue in the last version (21.5.1). It turns out that the metrics from the devices lost track until we restart the dispatcher container. We found a temporary solution adding a service that restarts all unhealthy containers affected by redis (we also have to create a new librenms image that allows this feature). Documentation about docker healthcheck.

Hope it helps.

[ottorei]

#13094 may possibly fix this

Edit: Did not fix the issue.

[ottorei]

This issue seems to be caused by unhandled exceptions on some Redis-functions. This also causes the nodes to stop reporting their stats when using the dispatcher service if they lose their connection to the Redis-instance.