Update Removes Tailscale: EdgeMAX EdgeRouter Firmware v2.0.9-hotfix.6

[yabdali]

Hi,
Reference to my previous issue, a new hotfix broke the installation. As suggested by you, I run the commands to check if the package still exists, attached are before and after the upgrade. Any hints?
https://github.com/jamesog/tailscale-edgeos/issues/14

Before Update:

After Update

[jamesog]

Hey @yabdali, thanks for that output, that's helpful. Do you still have this router in the broken state to be able to run some more commands?

The first thing I want to check is if the firstboot script maches the latest version in the repo. Most importantly I want to see if it has this block:

tailscale-edgeos/firstboot.d/tailscale.sh

Lines 82 to 96 in cb7f10c

	pkg_status=$(dpkg-query -Wf '${Status}' tailscale 2>/dev/null || true)
	if ! echo $pkg_status| grep -qF "install ok installed"; then
	# Sometimes after a firmware upgrade the package goes into half-configured state
	if echo $pkg_status | grep -qF "half-configured"; then
	# Use systemd-run to configure the package in a separate unit, otherwise it will block
	# due to tailscaled.service waiting on vyatta-router.service, which is running this script.
	systemd-run --no-block dpkg --configure -a
	else
	echo "Installing Tailscale"
	apt-get update
	apt-get install tailscale
	mkdir -p /config/data/firstboot/install-packages
	cp /var/cache/apt/archives/tailscale_*.deb /config/data/firstboot/install-packages
	fi
	fi

In either case, can you also run dpkg-query -Wf '${Status}' tailscale. I'm curious to see if this says something like half-configured, that the firstboot script currently checks for. My hunch is that the condition in the script isn't matching what's happening on your system. Tailscale looks installed but not configured properly.

Finally, running dpkg --configure -a should get that system up and running again.

[yabdali]

dpkg --configure -a

I tried sudo dpkg --configure -a but it doesn't seem to fix the issue, not found! I am attaching the scripts you mentioned for you to check.
firstboot-tailscale.txt

post-config-tailscale.txt

[yabdali]

I had to run the post config script and it installed tailscale again. Any suggestions for future upgrades?

[jamesog]

You could try enabling logs in systemd (EdgeOS turns them off by default to not cause wear on the flash drive):

set system systemd journal storage volatile

That will keep journald logs in memory, and are lost on reboot. But once a system has booted with this enabled it'd be interesting to see the output from

journalctl -u vyatta-router.service

I think the firstboot and post-config scripts should output there.

I'll leave this issue open now, as this is obviously a recurring issue, although weirdly I've only seen these kinds of issues on the ER-X.

[yabdali]

You could try enabling logs in systemd (EdgeOS turns them off by default to not cause wear on the flash drive):

set system systemd journal storage volatile

That will keep journald logs in memory, and are lost on reboot. But once a system has booted with this enabled it'd be interesting to see the output from

journalctl -u vyatta-router.service

I think the firstboot and post-config scripts should output there.

I'll leave this issue open now, as this is obviously a recurring issue, although weirdly I've only seen these kinds of issues on the ER-X.

I have set a remote syslog server to capture logs but it would only operate over Tailscale as the syslog server is at a different location. I will setup another log server at the location of the ER-X and will capture the logs on the next update. Thanks

[nnewc]

In my case, the old firmware image filled up my entire storage. I had to ssh into my EdgeRouter-X and manually delete the firmware image otherwise the tailscale install scripts would fail with 'no space left on device' errors. 😡

[altendorfme]

I had the same problem, when I updated to hotfix 6 everything stopped working, but I ran a delete system image and /config/scripts/post-config.d/tailscale.sh and everything was reinstalled correctly and I started with tailscale up --advertise-routes 192.168.1.0/24 --advertise-exit-node

Everything is working except the exit node via Edge Router :/

EDIT:

After restarting the Edge Router the exit node working correctly!

[yabdali]

I had the same problem, when I updated to hotfix 6 everything stopped working, but I ran a delete system image and /config/scripts/post-config.d/tailscale.sh and everything was reinstalled correctly and I started with tailscale up --advertise-routes 192.168.1.0/24 --advertise-exit-node

Everything is working except the exit node via Edge Router :/

EDIT:

After restarting the Edge Router the exit node working correctly!
I would suggest you change 192.168.1.0/24 to some other private subnet. I have faced issues when I am connecting more devices in networks that use this common subnet. Just an advice...

[jaredbileski]

having the same issues on an updated/wiped ER-X. Trying to run the firstbook script is returning this:

[jamesog]

That screenshot shows your router does not have the correct time: note the SSL error saying "certificate is not yet valid".

[nnewc]

Now seeing the device full error when there is no extra firmware image:

$ delete system image
The system currently has the following image(s) installed:

v2.0.9-hotfix.6.5574651.221230.1015 (running image) (default boot) 

There is no image to delete.

$ sudo apt install tailscale
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  tailscale
0 upgraded, 1 newly installed, 0 to remove and 2 not upgraded.
1 not fully installed or removed.
Need to get 0 B/21.8 MB of archives.
After this operation, 45.2 MB of additional disk space will be used.
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 37092 files and directories currently installed.)
Preparing to unpack .../tailscale_1.42.0_mipsel.deb ...
Unpacking tailscale (1.42.0) ...
dpkg: error processing archive /var/cache/apt/archives/tailscale_1.42.0_mipsel.deb (--unpack):
 cannot copy extracted data for './usr/bin/tailscale' to '/usr/bin/tailscale.dpkg-new': failed to write (No space left on device)
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Errors were encountered while processing:
 /var/cache/apt/archives/tailscale_1.42.0_mipsel.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

[nnewc]

Update: solved.

I already ran sudo apt purge tailscale, but I found an old tailscale .deb file in

$ rm /root.dev/w/var/cache/apt/archives/tailscale_1.36.1_mipsel.deb

[darthShadow]

May be worth trying out a few things from here (like upx & building custom binary) to get smaller binaries:

https://tailscale.com/kb/1207/small-tailscale/

[Wakeful-Cloud]

Ran into this issue with UPX 3.96-3, upgrading to 4.1.0 fixed it. Beyond that, it's pretty easy. You may need to modify build_dist.sh to use GOOS=linux GOARCH=mipsle.

[bartaspoz]

In my case, the old firmware image filled up my entire storage. I had to ssh into my EdgeRouter-X and manually delete the firmware image otherwise the tailscale install scripts would fail with 'no space left on device' errors. 😡

I had exactly the same issue on one of my Edgerouter-X. Now I have a second one that have a pending update and I'm sure that I will encounter the same issue. So once I will trigger the update I will loose access and this is a site where I have a limited physical access.
I'm wondering if we could embed somewhere in the tailscale install script the delete system image command?