rabbitmqctl's erlang cookie is not the same as $HOME/.erlang.cookie

[Zhan-Jie]

For Those Landing Here from Google/Baidu/etc

TL;DR

• See this doc section on the Erlang cookie
• See this rabbitmq-users thread where this has been answered
• If you don't mean to set the cookie via RABBITMQ_ERLANG_COOKIE (not the most secure option), make sure that env variable is not set since CLI tools will pick it up. Use the $HOME/.erlang.cookie file with 400 permissions or the --erlang-cookie switch.

Original Question

I've set docker container's environment variable HOME to /var/lib/rabbitmq。
After rabbitmq started, I could see its cookie hash like this :

node           : rabbit@rabbitmq-0.rabbitmq.default.svc.cluster.local.
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/rabbitmq/rabbitmq.conf
 cookie hash    : ilFMuK/7ghwxZmBkF0VDLg==
 log(s)         : /var/log/rabbitmq/rabbit@rabbitmq-0.rabbitmq.default.svc.cluster.local..log
                : /var/log/rabbitmq/rabbit@rabbitmq-0.rabbitmq.default.svc.cluster.local._upgrade.log
 database dir   : /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-0.rabbitmq.default.svc.cluster.local.

however, when I execute command rabbitmqctl -l status， it gave me following error:

Status of node rabbit@rabbitmq-0.rabbitmq.default.svc.cluster.local. ...
Error: unable to perform an operation on node 'rabbit@rabbitmq-0.rabbitmq.default.svc.cluster.local.'. Please see diagnostics information and suggestions below.

Most common reasons for this are:

 * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
 * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
 * Target node is not running

In addition to the diagnostics info below:

 * See the CLI, clustering and networking guides on http://rabbitmq.com/documentation.html to learn more
 * Consult server logs on node rabbit@rabbitmq-0.rabbitmq.default.svc.cluster.local.

DIAGNOSTICS
===========
attempted to contact: ['rabbit@rabbitmq-0.rabbitmq.default.svc.cluster.local.']

rabbit@rabbitmq-0.rabbitmq.default.svc.cluster.local.:
  * connected to epmd (port 4369) on rabbitmq-0.rabbitmq.default.svc.cluster.local.
  * epmd reports node 'rabbit' uses port 25672 for inter-node and CLI tool traffic 
  * TCP connection succeeded but Erlang distribution failed 

  * Authentication failed (rejected by the remote node), please check the Erlang cookie

Current node details:
 * node name: 'rabbitmqcli18@rabbitmq-0.rabbitmq.default.svc.cluster.local.'
 * effective user's home directory: /var/lib/rabbitmq
 * Erlang cookie hash: +MX//PtiDHrEvoWfQDsuuQ==

It seems that rabbitmqctl didn't read erlang cookie from $HOME/.erlang.cookie (i.e. /var/lib/rabbitmq/.erlang.cookie).
when I specify the erlang cookie explicitly and execute rabbitmqctl -l --erlang-cookie $(cat $HOME/.erlang.cookie) status， it gives me normal correct output...

I suspect this is a severe BUG

[Zhan-Jie]

rabbitmq-server version is 3.7.5

[michaelklishin]

Thank you for your time.

Team RabbitMQ uses GitHub issues for specific actionable items engineers can work on. GitHub issues are not used for questions, investigations, root cause analysis, discussions of potential issues, etc (as defined by this team).

We get at least a dozen of questions through various venues every single day, often light on details.
At that rate GitHub issues can very quickly turn into a something impossible to navigate and make sense of even for our team. Because GitHub is a tool our team uses heavily nearly every day, the signal/noise ratio of issues is something we care about a lot.

Please post this to rabbitmq-users.

Thank you.

[michaelklishin]

The cookie is read by the runtime, not RabbitMQ. Except for default location on Windows, nothing in that area has changed in many years. Make sure that the path is what you expect and that the file is readable. A number of other common reasons is listed right in the message. There's also a separate doc section on the subject.

[michaelklishin]

Authentication failed (rejected by the remote node), please check the Erlang cookie

is also a reason provided by the runtime. All the hints are pointing at the fact that the CLI tool (which might have been invoked as root or another user that's not used by the target node) uses a different cookie value from the target node.

[Zhan-Jie]

I would really appreciate it if you can answer one more question ; ) .

Why did the CLI tool use a different cookie value from the target node ? In addition to explicit command line option --erlang-cookie , how to indicate a given erlang cookie to the CLI tool ?

[michaelklishin]

Please take this to the mailing list. My main hypothesis is that the user used to invoke rabbitmqctl is different from the one used by the server. Which is fine but means that synchronising the cookie file is up to you.

[michaelklishin]

rabbitmq-users thread where this has been sorted.