Description
Is your feature request related to a problem? Please describe.
I ran into an edge case where celery was taking significantly longer than normal due to localhost not resolving locally.
Describe the solution you'd like
I'd like for Tactical to be fast at all times and resolve localhost without making a network request.
Describe alternatives you've considered
This if more of a discussion about a strange edge case and possible documentation in case others run into the same problem. I've considered not opening this request and making note of it for future reference but that doesn't help the community.
Additional context
I installed Ubuntu 20.04 in an nspawn container with debootstrap and then installed Tactical using the standard install. One asset was added. Everything was working as expected. Then I introduced Nebula and in the process changed the DNS to use a server in the cloud. This caused significant delays in the frontend but technically everything still worked.
The troubleshooting process led to a celery being slow and timing celery shows about a 5 second delay. Compare the production timing with the dev timing.
Production
time /rmm/api/env/bin/celery -A tacticalrmm list bindings
Queue Exchange Routing Key
---------------- ---------------- ----------------
celery celery celery
real 0m0.337s
user 0m0.306s
sys 0m0.028s
Dev
time /rmm/api/env/bin/celery -A tacticalrmm list bindings
Queue Exchange Routing Key
---------------- ---------------- ----------------
celery celery celery
real 0m5.726s
user 0m0.321s
sys 0m0.039s
The output of time /rmm/api/env/bin/celery -A tacticalrmm report showed redis://localhost/ was used which clued me into localhost. This lead to the question: Where is localhost resolved? A fresh Ubuntu install creates /etc/hosts and adds the various localhost flavors for name resolution.
Fresh install of Ubuntu 20.04 in VM
127.0.0.1 localhost
127.0.1.1 $hostname
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Comparing a fresh Ubuntu 20.04 install with an nspawn install, /etc/hosts is missing from the nspawn install and present in the full install. /etc/nsswitch.conf is the same for host resolution: hosts: files dns. This leads us to DNS. Both systems have /etc/resolv.conf symlinked to /run/systemd/resolve/stub-resolv.conf which is provided by systemd-resolved.service. The change I made that broke celery was changing the symlink to /run/systemd/resolve/resolv.conf and adding a DNS=1.1.1.1 entry to /etc/systemd/resolved.conf, effectively removing systemd-resolved from the equation.
The systemd-resolved documentation states "systemd-resolved' will resolve synthetic records, one of which is localhost. If localhost is not in /etc/hosts and the DNS has not been changed, localhost will still resolve locally without making any DNS requests over the network. However, if localhost is not in /etc/hosts and the DNS has been changed, there will be significant delays in the frontend of Tactical.
Adding localhost to /etc/hosts if it's not already present doesn't introduce any problems because the installer adds them. I suggest checking for localhost and if it's not already present, add it to /etc/hosts.