tunnel broker fails intermittently preventing nodes from connecting to the internet

[paidforby]

Mesh home nodes (mynet N600 or mynet N750) do not seem to be meshing with one another over their ad-hoc interface regardless of sudowrt firmware build (april build, new build) or version of makenode used (old commit, newest commit). Last I recall being able to mesh was in mid-September when working on battery powered, sneaker nodes.

To reproduce:

1. Flash two nodes with any build of sudowrt-firmware and run makenode
2. Connect one node to the internet via your home router.
3. Connect your computer to the other node via its "peoplesiopen.net " SSID
4. Attempt to ping a location on the internet, for example:

ping archlinux.org

or

ping 8.8.8.8

Expected result:

an internet connection

64 bytes from apollo.archlinux.org (138.201.81.199): icmp_seq=3 ttl=52 time=204 ms

Actual result:

no internet connection

ping: archlinux.org: Temporary failure in name resolution

or

PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
From 100.65.7.129 icmp_seq=1 Destination Net Unreachable

where 100.65.7.129 is the IP of the home node

Other observations:

pplsopen.net-node2node SSID is visible from computer WiFi list.

[jhpoelen]

@paidforby thanks for reporting this. This is consistent what I have seen during experimental mesh setups using n600/n750 over the last weeks.

[paidforby]

Issue was actually caused by exit node being in a 'weird' state? Can we document the solution an add it to the operator's manual?

[jhpoelen]

Yes!

It appears that the issue was caused by some sort of VPN issue on the exit node.

On Monday night, as I was debugging a home node mesh setup, I was able to mesh with local sudoroom nodes, but wasn't able to find a route to the internet. However, after @Juul applied some magic (VPN service restart on exit node?), my test nodes were able to ping public ip addresses. Also, an apparent related issue was resolved after this assumed reset (see https://sudoroom.org/pipermail/mesh/2017-December/002719.html).

Ideally we'd reproduce and then fix the issue, document in operator's manual like you suggested.

[jhpoelen]

I may have produced the issue as I was building the https://github.com/sudomesh/tunneldigger-lab . Steps to reproduce

1. dig a tunnel to exit node using tunneldigger client
2. stop / start a bunch of time

expected -
tunnel sessions go up and down, interface l2tp0 appears and disappears

actual -
initially, behavior is as expected, but after a couple of cycles, syslog on client machine has entries like -

Dec 18 22:24:54 lightgrey td-client: Performing broker selection...
Dec 18 22:25:16 lightgrey td-client: No suitable brokers found. Retrying in 5 seconds
Dec 18 22:25:21 lightgrey td-client: Performing broker selection...
Dec 18 22:25:41 lightgrey td-client: No suitable brokers found. Retrying in 5 seconds
Dec 18 22:25:46 lightgrey td-client: Performing broker selection...
Dec 18 22:25:55 lightgrey td-client: Got termination signal, shutting down tunnel

@Juul - can you confirm that broker on exit node is no longer allow for the digging of tunnels?

[paidforby]

I am seeing this same behavior from tunneldigger-lab. Also, I have some newly flashed home nodes that are not getting to the internet. Has the exit node been restarted since @jhpoelen reproduced the original issue?

[jhpoelen]

@paidforby thanks for reproducing ... it seems like we may be on to something. Perhaps the next step would be to setup a test exit node to reproduce. Note sure about the tunneldigger broker restart on exit node. Perhaps @Juul knows, he's operating the broker. This does strike me as something we should address sooner rather than later.

[jhpoelen]

After @Juul restarted the node, I was able to big tunnels again using https://github.com/sudomesh/tunneldigger-lab#digging-a-tunnel . This suggests that the tunnel broker hangs at some point after opening/closing tunnel cycles.

[paidforby]

Another outage of the exit node today. Discovered while following service guide and attempting to ping a locally hosted service, as in this use case.
Some observations come to mind:

1. Why is there only one exit node? What can we do to get another, backup exit node running?
2. Why do we need an exit node to route to services? Is there a better (read: more distributed way) of getting nodes to know about each over a WAN? How can we prevent the exit node from hindering development of local services?

To me, tunneldigger appears broken (at least the way we're using it) and ill-fit for "virtual meshing" to begin with (it's a start topology right? not very meshy). I am interested in exploring alternatives. Any suggestions?

[jnny]

Re: point 1: I can't answer with as much authority as @Juul (who'll probably weigh in later today), but it was my understanding we were switching to Hurricane Electric (HE) for the exit node and setting up another one at sudo room for backup purposes. Last I remember an update on the server at HE was in December, @jtremback and @Juul were coordinating setting it up. I believe there were 2 gigabit switches in sudo that we were going to set up on our rack up in sudo and in the HE cabinet.

there's also this guide to setting up an exit node that we should probably update: https://sudoroom.org/wiki/Mesh/Exit_setup

Re: point 2: I believe @papazoga was working on a tunneldigger alternative here: https://github.com/sudomesh/foutun but i'm not aware of its status.

side note: juul has been paying for the exit node and the dev droplet at ~$150/month out of pocket for about a year and a half now. Might be time to switch that cost over to sudomesh.

[bennlich]

Hey there. I think I am seeing the same bug on a home node that I recently flashed. I've connected the home node to my home router and am able to access the internet via the private SSID. I see two public SSIDs, peoplesopen.net 65.28.1 and peoplesopen.net 65.28.1 fast, neither of which provide internet access.

When I restart tunneldigger with /etc/init.d/tunneldigger restart, I see this in /var/log/messages:

root@pattyspuddles:~# tail -f /var/log/messages | grep td-client
Sun Feb  4 12:18:56 2018 daemon.warn td-client: Got termination signal, shutting down tunnel...
Sun Feb  4 12:19:06 2018 daemon.info td-client: Performing broker selection...
Sun Feb  4 12:19:09 2018 daemon.info td-client: Selected 45.34.140.42:8942 as the best broker.
Sun Feb  4 12:19:12 2018 daemon.info td-client: Tunnel successfully established.
Sun Feb  4 12:19:12 2018 daemon.info td-client: Requesting the broker to configure downstream bandwidth limit of 1000 kbps.

ping -I mesh5 8.8.8.8 times out (though I am not clear on whether that's the correct interface to use to test my connectivity to the mesh via the tunnel)
ping -I l2tp0 8.8.8.8 also times out (again, not positive if this ping should work even when tunneldigger is working as expected)

[jhpoelen]

Some days ago, tunneldigger issues became more pronounced, with all (?) node getting disconnected.

@Juul shared the following server config -

---

• babeld: HEAD https://github.com/sudomesh/babeld
• tunneldigger broker: https://github.com/sudomesh/tunneldigger commit f05d9adc170929f883600c3637b66b9c60705630
• tunneldigger client: HEAD from https://github.com/sudomesh/tunneldigger
• OS and kernel on tunneldigger broker server: Debian Jessie, kernel: 3.16.0-5-amd64

Something changed that broke things and I have no idea what. Possibly the kernel.

I tried switching the server to the new version of the tunneldigger server (tunneldigger broker) and after fixing a small bug it worked but only with the new version of the tunneldigger client.

With the new server and old client the server can send to the client over the tunnel but the client cannot send to the server.

With the old server even getting a tunnel fails most, but not all, of the time. Even when it succeeds no traffic can pass through it in either direction.

This could have been simple to troubleshoot if the server (written in python) was just creating these tunnels with calls to the ip l2tp command, but of course they implemented it using a netlink socket...

---