RabbitMQ 3.13.0 nodes with peer discovery enabled fails to start in an IPv6-only environment

[mnasiadka]

Describe the bug

In OpenStack Kolla-Ansible we support ipv6-only OpenStack deployments, including RabbitMQ.
The following config worked up until RMQ 3.13:

https://8dff850f95026878e224-be64ab2c1211c8399b7ac2efe4751a63.ssl.cf2.rackcdn.com/911094/2/check/kolla-ansible-rocky9-multinode-ipv6/4952d5c/primary/logs/kolla_configs/rabbitmq/rabbitmq.conf

https://8dff850f95026878e224-be64ab2c1211c8399b7ac2efe4751a63.ssl.cf2.rackcdn.com/911094/2/check/kolla-ansible-rocky9-multinode-ipv6/4952d5c/primary/logs/kolla_configs/rabbitmq/index.html

Logs and crash dump here: https://8dff850f95026878e224-be64ab2c1211c8399b7ac2efe4751a63.ssl.cf2.rackcdn.com/911094/2/check/kolla-ansible-rocky9-multinode-ipv6/4952d5c/primary/logs/kolla/rabbitmq/index.html

Reproduction steps

1. Use config from description to start RMQ 3.13

...

Expected behavior

Runs correctly

Additional context

No response

[dumbbell]

Hi! Thank you for the report.

How do you specify the path to the inetrc file?

The peer discovery subsystem starts a temporary intermediate node to query some properties of remote nodes. I suspect that this temporary node doesn't use this inetrc file.

[mnasiadka]

Here is the container log: https://8dff850f95026878e224-be64ab2c1211c8399b7ac2efe4751a63.ssl.cf2.rackcdn.com/911094/2/check/kolla-ansible-rocky9-multinode-ipv6/4952d5c/primary/logs/container_logs/rabbitmq.txt

We just put erl_inetrc under /etc/rabbitmq/erl_inetrc with this content: https://8dff850f95026878e224-be64ab2c1211c8399b7ac2efe4751a63.ssl.cf2.rackcdn.com/911094/2/check/kolla-ansible-rocky9-multinode-ipv6/4952d5c/primary/logs/kolla_configs/rabbitmq/erl_inetrc

Is there something more we should be doing?

[dumbbell]

No, this is a limitation in the way that temporary node is started probably. I will try to reproduce locally.

[lukebakken]

I suspect that this temporary node doesn't use this inetrc file

I'm sure you're right @dumbbell

[michaelklishin]

This may be worth delaying 3.13.1 over.

[dumbbell]

I'm starting to look at this issue today, bu I can't find where this /etc/rabbitmq/erl_inetrc file is configured. RabbitMQ itself doesn't look at a default location, so it must come from something outside of RabbitMQ.

Could you please share the output of the following two commands?

• rabbitmqctl eval 'os:getenv().'
• rabbitmqctl eval 'application:get_all_env(kernel).'

[artificial-intelligence]

we set this via the environment, see the jinja template here: https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/rabbitmq/templates/rabbitmq-env.conf.j2 (line 10)

I don't currently have shell access to the CI env where this fails, but as you can see from the git history this part of the code didn't change since 2019, so here are the commands you wanted the output from, from a similar environment, but with a slightly older rmq release.
you can see the erl_inetrc is loaded there as well, I just edited out some stuff like hostnames and IPs:

rabbitmqctl eval 'os:getenv().'
["KOLLA_DISTRO_PYTHON_VERSION=3.10",
 "RABBITMQ_CLUSTER_COOKIE=REDACTED",
 "RABBITMQ_LOG_DIR=/var/log/kolla/rabbitmq","ROOTDIR=/usr/lib/erlang",
 "SHLVL=0","SYS_PREFIX=","RABBITMQ_NODENAME=rabbit@REDACTED",
 "RABBITMQ_PID_FILE=/var/lib/rabbitmq/mnesia/rabbitmq.pid",
 "RABBITMQ_HOME=/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.11.23",
 "RABBITMQ_LOG_BASE=/var/log/kolla/rabbitmq","RABBITMQ_DIST_PORT=25672",
 "OLDPWD=/","PROGNAME=erl","PWD=/var/lib/rabbitmq",
 "PATH=/usr/lib/erlang/erts-13.2.2.3/bin:/usr/lib/erlang/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
 "PLUGINS_DIR=/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.11.23/plugins",
 "LANG=en_US.UTF-8","MNESIA_BASE=/var/lib/rabbitmq/mnesia",
 "KOLLA_SERVICE_NAME=rabbitmq","HOME=/var/lib/rabbitmq",
 "KOLLA_BASE_DISTRO=ubuntu","KOLLA_CONFIG_STRATEGY=COPY_ALWAYS",
 "HOSTNAME=REDACTED","KOLLA_BASE_ARCH=x86_64","ERL_EPMD_ADDRESS=REDACTED",
 "ERL_INETRC=/etc/rabbitmq/erl_inetrc","ERL_MAX_ETS_TABLES=50000",
 "ERL_MAX_PORTS=65536",
 "ERL_LIBS=/usr/lib/rabbitmq/lib/rabbitmq_server-3.11.23/plugins",
 "ERL_EPMD_PORT=4369","DEBIAN_FRONTEND=noninteractive",
 "ENABLED_PLUGINS_FILE=/etc/rabbitmq/enabled_plugins",
 "ERL_CRASH_DUMP=/var/log/kolla/rabbitmq/erl_crash.dump","EMU=beam",
 "BINDIR=/usr/lib/erlang/erts-13.2.2.3/bin",
 "CONFIG_FILE=/etc/rabbitmq/rabbitmq",
 "ADVANCED_CONFIG_FILE=/etc/rabbitmq/advanced.config"]

and the other command (your command missed a ' at the end):

 rabbitmqctl eval 'application:get_all_env(kernel).'
[{shell_docs_ansi,auto},
 {net_ticktime,60},
 {shutdown_func,{rabbit_prelaunch,shutdown_func}},
 {inet_default_connect_options,[{nodelay,true}]},
 {connect_all,true},
 {logger_level,notice},
 {inet_dist_listen_min,25672},
 {inetrc,'/etc/rabbitmq/erl_inetrc'},
 {logger_sasl_compatible,false},
 {logger,
     [{handler,default,logger_std_h,
          #{config => #{type => standard_io},
            formatter =>
                {logger_formatter,
                    #{legacy_header => true,single_line => false}}}}]},
 {inet_dist_listen_max,25672},
 {inet_dist_use_interface,{REDACTED}},
 {net_tickintensity,4},
 {prevent_overlapping_partitions,false}]

HTH in the debugging process.

[dumbbell]

Thank you very much! Yes, it definitely helps.

I fixed the command in my comment, thanks.