Some Nodes show as relaying after a few hours?

[seanbts]

Hi,

I have your software running in docker on a VPS and have 40 Teltonika RUT240 connected all via 4g with the same provider, all RUT240's have the same setup, after a few hours some of the units are showing as relaying, if i restart the VPS running zero-ui, they all go back to online, any issues how to resolve to make sure all nodes stay online and not relay?

Many Thanks.

[dec0dOS]

Hello, @seanbts. It seems to be the issue with the ZeroTier controller itself and networking, but not the ZeroUI. 4g connections often have problems with P2P services.

There is an issue with updating the Docker image of the controller to the latest version, you could try setting up the latest version of the ZeroTier controller on the host (check the docs for more info).

[timothyl13241]

Hi @seanbts,

Thank you for sending the screenshots. Which VPS are you using? I ask because I had a similar thing happen when I was hosting the ZT controller on Google Cloud. When I switched over to Oracle Cloud and assigned both a public IPv4 and IPv6 address to the VPS, I no longer had all of my nodes displaying RELAYED status.

I believe within 'zerotier-cli peers', the field updates more slowly than the (latency) field. Typically if a client has a -1 latency, there is something going on with the link between the controller and the peer.

To further check, are you able to run 'zerotier-cli peers' on other nodes on the same network and post the results?

[seanbts]

Thanks @timothyle97

The VPS is a digitalocean droplet, I just checked and IPv6 wasn't enabled, so i've enabled it now and confirmed IPv6 is working, should i open port 9993 on the VPS UFW FW for both IPv4 and IPv6? Could having the IPv6 not enabled been a cause of the relay issue? I've rebooted the VPS now and will monitor over the next few hours. I'd also be interested if you have a recommendation for a VPS that has worked for you.

Thanks again.

[timothyl13241]

Hi @seanbts,

Is your VPS behind NAT or does it have a direct public IP? Just trying to think of the different things that could be causing the issue.

Looking at your previous screenshots, I don't think IPv6 is a direct cause of clients becoming relayed, but it could contribute to the issue. I have only tried Google Cloud and Oracle Cloud. For some reason I found that Oracle gave me fewer issues with relayed clients.

I have been trying to find a concrete answer to whether port 9993/UDP is required inbound, but I have not been able to find much information on this. I believe that it is possible for the ZT controller to "punch holes" in the firewall even if port 9993/UDP inbound is blocked, but I have left 9993/UDP inbound for both Google Cloud and Oracle Cloud. I have tried a port scan on 9993 on one of my ZeroTier Central controllers and I found port 9993 open. Whether this was transient or static I cannot tell for sure.

Let me know if your peers continue to go offline after some time.

[dec0dOS]

Regarding the 9993 port: you should block the 9993/TCP inbound and outbound if you're running the ZeroTier controller on the host (not in the docker), but you should allow the 9993/UDP inbound and outbound to make the ZeroTier work correctly.

9993/TCP is used to expose the ZeroTier controller API, but it is not meant to be used over the WLAN. ZeroUI backend proxifies the requests to the ZeroTier controller API. 9993/UDP is the main port that is used for ZeroTier P2P connections.

More info about the network controller could be found here: https://github.com/zerotier/ZeroTierOne/tree/master/controller

[seanbts]

Thanks @timothyle97 @dec0dOS , so far nearly 24 hours on after enabling IPv6 on the VPS ( which has public accessible IPv4 and IPv6 ), all 44 peers are direct and haven't gone to relaying, will monitor over the next few days and report any changes, appreciate all the guidance.
Cheers

[seanbts]

Hi @dec0dOS , So after a couple of days, we now have a lot of peers relaying again, i suspect it could be down to VPS hosting, so as suggested by @timothyle97 , i've setup a VPS on Oracle and have Zero-UI installed, what's the best way to backup and restore our existing data so i can migrate over to Oracle VPS?

Many Thanks

[dec0dOS]

@seanbts check the README backup section

[timothyl13241]

@seanbts Let me know if you run into any issues with the migration.

Since ZeroTier is installed directly to your previous VPS, I believe you can install ZeroTier on the new VPS, then stop the service, and copy the entire zerotier directory (including controller.d, identity.secret, identity.public, etc) to the new VPS. Make sure that ZeroTier is shut down on your previous VPS and then restart the zerotier service on the new VPS.

[seanbts]

Hi @dec0dOS , I spun up a VPS on Oracle using Ubuntu and a ARM Instance, when i issue docker-compose pull, i get the following error, is this related to the fact that it's ARM technology?

The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
standard_init_linux.go:228: exec user process caused: exec format error

[dec0dOS]

Hello, @seanbts. Currently, there is no prebuild linux/arm64/v8 image. You should use the linux/arm/v7 image instead. Strangely, docker does not pull this image automatically.
Try the following commands:

docker pull --platform linux/arm/v7 dec0dos/zerotier-controller:latest
docker pull --platform linux/arm/v7 dec0dos/zero-ui:latest

Alternatively, you could build the images from the source by yourself. Clone the repo and use the docker-compose build command (check the following threads for more info: #28 (reply in thread), #1 (reply in thread))

[seanbts]

Cheers @dec0dOS , done that and it pulled those - If i then run docker-compose up -d --no-build, i get the same error, is there a specif command for bringing up the containers?

[dec0dOS]

That is strange that docker does not want to use the linux/arm64/v7 image on the linux/arm64/v8 machine. Thanks for your report, in the next few releases I'll build the specific image for the linux/arm64/v8. But for now, you could build the linux/arm64/v8 from the source.

[timothyl13241]

@seanbts, just wondering if you had success migrating to Oracle Cloud and if you have been able to test with it.