Description
Thanks to https://peoplesopen.net/monitor it is now easier to track this. See #8 for early observations.
The Bug
On a fresh boot of the psychz exit node, home nodes dig tunnels, babel babels, and everybody's routing tables get filled with mesh routes. But...
Over time (after about 24-48 hours), routes start to slowly disappear from the routing table, and they don't return until babeld and tunneldigger-broker are restarted on the exit node.
Debugging
This appears to be due to a memory leak in babeld. When the exit node is in the bad state, looking at /var/log/babeld.log during a tunnel connect shows:
Warning: cannot save old configuration for l2tp4061.
setsockopt(IPV6_JOIN_GROUP): Cannot allocate memory
setsockopt(IPV6_LEAVE_GROUP): Cannot assign requested address
Warning: cannot restore old configuration for l2tp4061.
i.e. babeld tries to add the socket to its ipv6 broadcast group and fails due to a memory allocation error.
When the exit node is in a healthy state, no such errors get logged to /var/log/babeld.log, and the mesh routes get added to the routing table as expected.
Conclusion
It looks like there's a socket option memory leak in babeld. I think we're only seeing this bug now in the last month because someone happens to be running a weird node that disconnects and reconnects its tunnel every 5 minutes. You can see this behavior by watching /var/log/syslog on the psychz node for 5 minutes.
Every time the rogue node destroys and recreates a tunnel, the tunneldigger up and down hooks are run, the old tunnel interface is removed from babeld (babeld -x $ifname) and the new tunnel interface is added (babeld -a $ifname).
It seems that removing an interface from babeld does not properly clean up all used memory, and eventually babeld is unable to setsockopt on new sockets.
Todo
Look into socket option memory allocation? Halp!