rabbitmqctl in combination with ssl

[Thubo]

Hi there,

I'm trying to run the rabbitmq container with ssl/tls.

So far I got ssl/tls to work (i.e. a connection to the server is possible, messages are delivered, clients verified, etc.) - it's all good on that side.

However as soon as I try to manage the node with rabbitmqctl I get the following error:

# docker exec -it rabbit-00 rabbitmqctl list_users
Error: unable to connect to node 'rabbit@rabbit-00': nodedown

DIAGNOSTICS
===========

attempted to contact: ['rabbit@rabbit-00']

rabbit@rabbit-00:
  * connected to epmd (port 4369) on rabbit-00
  * epmd reports node 'rabbit' running on port 25672
  * TCP connection succeeded but Erlang distribution failed

  * Remote host closed TCP connection before completing authentication. Is the Erlang distribution using TLS?


current node details:
- node name: 'rabbitmq-cli-34@rabbit-00'
- home dir: /var/lib/rabbitmq
- cookie hash: blabla

I already tried the docker image cvtjnii/rabbitmq:3.6.1 which showed up in #69, which works fine.

(Note that using the /docker-entrypoint.sh as a wrapper does not work with this official image:

# docker exec -it rabbit-00 /docker-entrypoint.sh rabbitmqctl list_users
Error: unable to connect to node 'rabbit@rabbit-00': nodedown

DIAGNOSTICS
===========

attempted to contact: ['rabbit@rabbit-00']

rabbit@rabbit-00:
  * connected to epmd (port 4369) on rabbit-00
  * epmd reports node 'rabbit' running on port 25672
  * TCP connection succeeded but Erlang distribution failed
  * suggestion: hostname mismatch?
  * suggestion: is the cookie set correctly?
  * suggestion: is the Erlang distribution using TLS?

current node details:
- node name: 'rabbitmq-cli-52@rabbit-00'
- home dir: /var/lib/rabbitmq
- cookie hash: blabla

Here is my startup script - as I said, the TLS connection itself via port 5671 is working fine, but the rabbitmqctl is not (which makes clustering and administration a little difficult).

#!/bin/bash

ERLANG_COOKIE="cookie"

SSL_LISTENERS="5671"
SSL_CACERTFILE="/certs/ca/cacert.pem"
SSL_CERTFILE="/certs/server/rabbit-00.vagrant.cert.pem"
SSL_KEYFILE="/certs/server/rabbit-00.vagrant.key.pem"
SSL_FAIL_IF_NO_PEER_CERT="false"
SSL_VERIFY="verify_peer"

docker run \
  -d \
  --restart=always \
  --name rabbit-00 \
  --hostname rabbit-00.vagrant \
  -p 4369:4369 \
  -p 5671:5671 \
  -p 5672:5672 \
  -p 15671:15671 \
  -p 15672:15672 \
  -p 25672:25672 \
  -e RABBITMQ_ERLANG_COOKIE=$ERLANG_COOKIE \
  -e RABBITMQ_SSL_CERT_FILE=$SSL_CERTFILE \
  -e RABBITMQ_SSL_KEY_FILE=$SSL_KEYFILE \
  -e RABBITMQ_SSL_CA_FILE=$SSL_CACERTFILE \
  -e RABBITMQ_SSL_FAIL_IF_NO_PEER_CERT=$SSL_FAIL_IF_NO_PEER_CERT \
  -e RABBITMQ_SSL_VERIFY=$SSL_VERIFY \
  -v /srv/docker/rabbit/certs:/certs \
  rabbitmq:3-management

Any help is appreciated. Thanks a lot!
Matthias

P.S. I already tried experimenting with the RABBITMQ_CTL_ERL_ARGS as in docker-entrypoint.sh#L277 but I had no luck so far. Maybe I'm just using it wrong...

[michaelklishin]

rabbitmqctl's inability to contact a node has nothing to do with whether TLS listener for client connections is used. Key part of the error is

rabbit@rabbit-00:
  * connected to epmd (port 4369) on rabbit-00
  * epmd reports node 'rabbit' running on port 25672
  * TCP connection succeeded but Erlang distribution failed

So rabbitmqctl can connect but cannot authenticate with rabbit-00. In 99% of cases it comes down to

• An Erlang cookie mismatch
• A hostname mismatch between what rabbitmqctl uses and what the node in question uses

[Thubo]

Thanks for the comment, but it's working without any modification using the cvtjnii/rabbitmq:3.6.1. As soon as I disable TLS (by not exporting the RABBITMQ_SSL_* variables everything works as well.

[michaelklishin]

@Thubo if inter-node links was configured to use TLS (which may or may not be the case, I don't follow these images closely), then rabbitmqctl has to be configured as well. I cannot think of any other way in which TLS can prevent rabbitmqctl from authenticating. In any case, see server logs. Guessing is a highly inefficient debugging technique.

[Thubo]

Frankly said: I don't get it ;) The docs you send me helped a little, but didn't fix the problem.

If I look a the logs, I see

=ERROR REPORT==== 17-Aug-2016::13:46:10 ===
Cannot accept TLS distribution connection: Invalid certificate file '/tmp/combined.pem': no such file or directory

when executing rabbitmqctl (both with and without RABBITMQ_CTL_ERL_ARGS as described in the docs).
The file /tmp/combined.pem is there, readable by rabbitmq:rabbitmq and populated as described in the docs.

---

For now I disabled the docker-entrypoint.sh#L265-L278 and rebuild the image.

Now both:

root@rabbit-00:/# export RABBITMQ_USE_LONGNAME=true
root@rabbit-00:/# rabbitmqctl -n rabbit@rabbit-00.vagrant list_users
Listing users ...
guest   [administrator]

and

# docker exec -it rabbit-00 /docker-entrypoint.sh rabbitmqctl list_users
Listing users ...
guest   [administrator]

work.

Exporting the USE_LONGNAME variable via docker run -e RABBITMQ_USE_LONGNAME=$USE_LONGNAME works as well and the export/docker-entrypoint wrapper are not needed. (Found here although I'm not sure the issues are related.

However this is not a real fix, but a quick and dirty work-around.

---

I would be happy if someone with more experience in Erlang/RabbitMQ clustering could look into this.

Can we keep the issue open until I/someone finds a permanent solution (which works out-of-the-box? Or somebody can confirm that it's just me and/or my setup which is not working as expected...

[basz]

@michaelklishin I have what i think is the exact same issue, it works until I enable tls. the web admin works as expected with tls.

could you show me what you commented exactly in https://github.com/docker-library/rabbitmq/blob/master/3.6/alpine/docker-entrypoint.sh as your link is now 404.

[basz]

I got it to work eventually simply by defining some attributes for erlang.

somehow it needs be told about using TLS

see #165