upgrade/ retire psychz exit node

[jhpoelen]

As discussed in #8 (comment) and related threads, the current installation of the psychz exit node has to run an older version of Debian (without security patches) in order to allow unpatched home nodes to create exit node tunnels.

At time of writing (11 April 2018) 15 home nodes are connecting to the psychz exit node. These nodes need to be upgrade before being able to upgrade or retire this exit node.

If the exit node is upgraded/ retired before that, the home nodes will be unable to create tunnels without physical access to the nodes. This means that the home nodes would not able to provide open internet access unless they mesh with neighboring nodes with a route to the internet.

[bennlich]

@Juul is planning to not renew psychz subscription very soon, so it's time to retire the exitnode.

IPs connected to psychz at time of posting:

• 100.64.55.192/26
• 100.64.63.0/26
• 100.65.14.64/26
• 100.65.140.128/26
• 100.65.20.0/26
• 100.65.6.0/26

[paidforby]

@jhpoelen is there anything that should be saved/rescued from the psychz server before it goes away forever? Not sure if there are any old configurations that might be worth backing up.

[jhpoelen]

great to hear that the psychz exit node is getting retired - it took us quite some work to be able to do this without bringing the network down and losing the connection to our existing nodes.

Some suggestions:

1. I believe that @bennlich made some recent updates to the exitnode configuration to quickly recover from suspected babeld bugs, however I saw no changes to https://github.com/sudomesh/exitnode . Suggest to include the config changes in the repository.
2. using https://github.com/sudomesh/exitnode script, instantiate at least one extra exitnode in addition to the he exitnode using an ip address that is under our control and has been installed as default on all existing home nodes. In my opinion, history has shown that having only a single exit node increases the risks of bricking home nodes due to a malfunctioning, or no longer available (due to blacklisting), exit node.

[paidforby]

I think work might have been done on that in the upgrade babeld branch and has been an open PR for sometime, sudomesh/exitnode#15, which also includes the ability to redeploy the exitnode by re-running create_exitnode.sh.

Also, maybe before psychz is turned off, we should test rebuilding HE by running create_exitnode.sh, just in case it goes horribly, irreparably wrong?

[paidforby]

Also, good idea about at least deploying the exitnode script on one of the backup servers. For reference, here's the potential servers that could be used, https://github.com/sudomesh/sudowrt-firmware/blob/master/files/opt/mesh/templates/etc/config/tunneldigger not sure about the configuration on any of them other than builds.sudomesh.org, which already has a ton of things running/stored on it.

[jhpoelen]

re: builds.sudomesh.org - I imagine that the dns would be relatively easy to point to another machine
to make room for a secondary exit node, compared to the scenario in which each node needs patching. On a related topic - have you considered moving the build / web server stuff to a dedicated non-do/non-heroku server? Perhaps worth noting that I recently rented a 16GB / 1TB server for $20 /mo with Hetzner.de , so there's much cheaper alternatives to the likes of heroku/ digital oceans, especially for running continuous running / heavy processes.
re: pending pull requests - I am used to a more continuous integration / deployment workflow, so I am not in the habit of checking the pending pull requests. Thanks for pointing out this pending request.

[paidforby]

good points about builds.sudomesh.org! Committing to moving builds.sudomesh.org would encourage us to cleanup that server and make sure everything on it is update to date, documented, and re-deployable. I like that idea so much that I'm going to open a new bug to document services currently running on it.

And agreed on PRs, I almost forgot that one existed, I encourage @bennlich to go ahead and merge it.

[paidforby]

So after copying the sudomesh server to a new droplet and new IP address, 107.170.221.27, I've realized that there is no way to decrease the size of the old sudomesh server, which way bigger than it needs to be for a back up exit node, and keep the old IP address, 107.170.219.5. @jhpoelen or @bennlich any ideas on how to preserve our precious IP address and downsize the droplet, I can't seem to find any way of doing this in digital ocean.

[bennlich]

Yep, I think you're right. From https://www.digitalocean.com/docs/droplets/how-to/resize/

CPU and RAM only resizes can be reversed if you want to return to using a smaller Droplet. Disk, CPU, and RAM resizes cannot be reversed because decreasing the size of the disk poses data integrity issues. Any time you resize a Droplet, be sure to test that all your services are running as expected.

In retrospect, we should have created a "digitalocean floating ip", which is a static ip address that you can move around between droplets, and use that IP in our tunneldigger configs. Woops.

Let's do this and patch some nodes? Doesn't really make sense to leave ourselves tied to a droplet with specs we don't really want IMO.

[jhpoelen]

From my patching experience last year, I've learned that patching nodes is a risky, time intensive process that I wouldn't wish on anyone to have to repeat. Rather that spending time on this, I'd favor to keep the droplet around, install tunnneldigger on it, and spent the time to work on ways to either get rid of exit nodes or figure out a more dynamic approach to connect to exit nodes. Meanwhile, would it be worth trying to convince digitalocean to turn our existing ip into one of these floating IPs?

[paidforby]

I opened a ticket with DO asking them to convert the static IP to a floating IP. We'll see what they say.

Here's a question, is there any reason we couldn't run all the sudomesh server and exit node services on the same droplet? I know it's not ideal, but it seems like the only solution at this point. I second @jhpoelen on exploring new ideas of how to host exit nodes, or if they are even necessary.