Issues with Perfsonar Toolkit on Ubuntu 20.04 with IPv6,(includes also fixes that i made)

[Rezi1999]

I was working on installing perfsonar-toolkit on ubuntu 20.04 with ipv6 only. To begin with, there were no errors during the installation, everything installed successfully. But several types of errors were observed on the web interface.

As for the first error, on the main page we have a working link of the perfsonar elmond archive Service, but when going to it, there was a dns proxy localhost error. After that, I did some troubleshooting, and I saw that this service was listening on port 5000, which is backed by the Apache webserver, and was proxying this service to localhost. The problem is with localhost, as I found out and tested it myself, ubuntu 20.04 by default has an entry that assigns localhost 127.0.0.1 ipv4 address, and we are using ipv6 at the moment, so this proxy will not work on localhost, because localhost returns ipv4 address. Therefore, I manually added an entry to the /etc/hosts file, where I wrote the ipv6 address to the localhost(::1).

I think this is a bug on Ubuntu 20.04 not to have ipv6 assigned to localhost, on ubuntu 22.04 for example its there so i assume its a bug on ubuntu 20.04 after that this dns proxy localhost error was fixed, but the service still did not work, it said service unavaliable. After that, I dug further and found the python script responsible for starting this service, which is the /usr/lib/perfsonar/elmond/app.py. I opened the script and on the last line I saw the following entry create_app().run(debug=true, host='127.0.0.1' which means that this service starts directly on 127.0.0.1 only, which will not work in case of ipv6, so i manually changed 127.0.0.1 to ::1 and it started working so to fix this issue developers should include in script both 127.0.0.1 and ::1 records to work as for ipv4 and ipv6.

Next issue was that perfsonar toolkit configdaemon service wasnot working because of same issue as we had on elmond service, in /etc/perfsonar/toolkit/configdaemon.conf file it was listening to 127.0.0.1 and i changed it to ::1 and it worked. also when i wanted to test the latency and bandwidth between the hosts, on web gui for that I used perfsonar's public hosts with ipv6 only, but when adding the host, it said that I could not save the configuration file, i couldnot fixed that issue its still relevant, i checked apache and toolkit logs but nothing is there.

this is a elmond service which was not working:

you can see here that apache proxys to locahost:

as you can see there is not Ipv6 record for localhost:

here i added manually:

here in this script /usr/lib/perfsonar/elmond/app.py you can see that script is executing only on 127.0.0.1:

here manually changed it to ipv6:

same goes for configdaemon service, it was listening on 127.0.0.1:

changed it to ipv6, and service started:

also there is was problem i couldnot add host from toolkit, there is an error "Couldn't save configuration file":

That issue was caused that client still was using ipv4 to contact daemon in code: /usr/share/perl5/perfSONAR_PS/NPToolkit/ConfigManager/Utils.pm

and i changed it to ipv6:

I installed all operating systems which supports perfsonar to check content of /etc/hosts for ipv6 localhost entry like this:
::1 localhost

Ubuntu 22
Ubuntu 20
Debian 12
Debian 11
Debian 10
Alma 9
Centos 7

Localhost entries are different between Ubuntu and Debian. if you install VM with ipv4 on Ubuntu 22 you have got ipv6 localhost entry in /etc/hosts file, but if you install with ipv6 you dont have it. On Debian you have it on both cases.

on Ubuntu 20 whether you install it with ipv4 or ipv6 you dont have ipv6 localhost record

[laeti-tia]

The IPv6 logstash listening issue is also mentioned in perfsonar/archive#23

[laeti-tia]

After the latest corrections, it seems only 2 services are not working fine when installing on an IPv6 only host: elmond and node_exporter.

@Rezi1999 might have more information.

[arlake228]

I just made a change to node_exporter in 5.1.1 that tells it to explicitly listen on localhost. Before it was just doing whatever the default was which led to it listening on all ports (maybe 0.0.0.0?). It would be worth trying again with new setting.

[Rezi1999]

I just made a change to node_exporter in 5.1.1 that tells it to explicitly listen on localhost. Before it was just doing whatever the default was which led to it listening on all ports (maybe 0.0.0.0?). It would be worth trying again with new setting.

I tried and its listening on IPv6, but the problem is regarding host exporter service, in configuration file /usr/lib/perfsonar/host_metrics/perfsonar_host_exporter it listens on localhost but when i check it with nestat command it shows that its listening on IPv4.

attaching screenshot:

netstat output:

[laeti-tia]

After looking more closely at this with @Rezi1999 it seems we're hitting a 10-year-old Python bug, probably both with perfsonar_host_exporter and elmond. Prometheus node_exporter itself is fine.

Problematic code in upstream Python: https://github.com/python/cpython/blob/06a1c3fb24c4be9ce3b432022ebaf3f913f86ba7/Lib/socketserver.py#L440

Bug reported under python/cpython#64414

I guess, as you suggested @arlake228, that the solution is to put both Python services behind the Apache mod_wsgi module.

[arlake228]

WSGI changes have been merged into 5.2.0 branch, so should be able to test this again.