Crash with root-owned Self-signed SSL Certificates

[mancvso]

As the script does not persist ERL_SSL_PATH, RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS neither RABBITMQ_CTL_ERL_ARGS, they have to be entered in a subshell for use with rabbitmqctl

But when trying to see status, it crash with ssl related messages (which I cannot decipher)

Certificates were created following instructions from Rabbit Website

Installing the CA in the machine/container does not helps either.

Running the rabbitmq server as root, solves the issue, but is far from optimal.

I've made a script so you can watch it die 😀

#!/bin/bash
mkdir testca
cd testca
mkdir certs private
chmod 700 private
echo 01 > serial
touch index.txt

cat > "openssl.cnf" <<EOF
[ ca ]
default_ca = testca

[ testca ]
dir = .
certificate = \$dir/cacert.pem
database = \$dir/index.txt
new_certs_dir = \$dir/certs
private_key = \$dir/private/cakey.pem
serial = \$dir/serial

default_crl_days = 7
default_days = 365
default_md = sha1

policy = testca_policy
x509_extensions = certificate_extensions

[ testca_policy ]
commonName = supplied
stateOrProvinceName = optional
countryName = optional
emailAddress = optional
organizationName = optional
organizationalUnitName = optional

[ certificate_extensions ]
basicConstraints = CA:false

[ req ]
default_bits = 2048
default_keyfile = ./private/cakey.pem
default_md = sha1
prompt = yes
distinguished_name = root_ca_distinguished_name
x509_extensions = root_ca_extensions

[ root_ca_distinguished_name ]
commonName = hostname

[ root_ca_extensions ]
basicConstraints = CA:true
keyUsage = keyCertSign, cRLSign

[ client_ca_extensions ]
basicConstraints = CA:false
keyUsage = digitalSignature
extendedKeyUsage = 1.3.6.1.5.5.7.3.2

[ server_ca_extensions ]
basicConstraints = CA:false
keyUsage = keyEncipherment
extendedKeyUsage = 1.3.6.1.5.5.7.3.1
EOF

openssl req -x509 -config openssl.cnf -newkey rsa:2048 -days 365 -out cacert.pem -outform PEM -subj /CN=MyTestCA/ -nodes
openssl x509 -in cacert.pem -out cacert.cer -outform DER
cd ..
ls
mkdir server
cd server
openssl genrsa -out key.pem 2048
openssl req -new -key key.pem -out req.pem -outform PEM -subj /CN=$(hostname)/O=server/ -nodes
cd ../testca
openssl ca -config openssl.cnf -in ../server/req.pem -out ../server/cert.pem -notext -batch -extensions server_ca_extensions
cd ..
ls
cp testca/cacert.pem server/cacert.pem
CWD=$(pwd)
ID=$(docker run -d --name dyingrabbit -v $CWD/server:/usr/local/share/certificates -e RABBITMQ_ERLANG_COOKIE=muchsecret -e RABBITMQ_SSL_CERT_FILE=/usr/local/share/certificates/cert.pem -e RABBITMQ_SSL_KEY_FILE=/usr/local/share/certificates/key.pem -e RABBITMQ_SSL_CA_FILE=/usr/local/share/certificates/cacert.pem -e RABBITMQ_USE_LONGNAME=true -h my.long.host.tld rabbitmq:3-management)
sleep 5
docker logs $ID
docker exec $ID bash -c "export ERL_SSL_PATH=\"/usr/lib/erlang/lib/ssl-7.1/ebin\" && export RABBITMQ_CTL_ERL_ARGS=\"-pa '/usr/lib/erlang/lib/ssl-7.1/ebin' -proto_dist inet_tls -ssl_dist_opt server_certfile /tmp/combined.pem -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true\" && rabbitmqctl status"
docker logs $ID

The intention is to enable a 3 machine cluster comunicating with SSL.

[mancvso]

I want to clarify that the issue is not about permissions with the certificates or the CA not being trusted by the base OS.

[hlm]

@mancvso I had to make some minor updates to reproduce the error. First I had to included a RABBITMQ_DEFAULT_USER var (keep reading to understand why) then I had to force the version to 3.6.0, not sure why, it could be something with my setup, but using the 3-management tag did not pick up the newer SSL handling code in docker-entrypoint.sh

The reason for including the RABBITMQ_DEFAULT_USER var can be explained if you look at docker-entrypoint.sh#L47. You should notice that a new /etc/rabbitmq/rabbitmq.config file will only be generated if the haveConfig var is true-ish, which only happens if you have one of the three DEFAULT variables set: RABBITMQ_DEFAULT_{VHOST,USER,PASS}.

I think it is fair to expect that the config file gets generated when any of the SSL env vars are given, even without the presence of any DEFAULT var, and, maybe subject for a broader discussion, docker-entrypoint.sh needs to be updated to conform to such behavior, it that is really the case.

Once you give at least one of the expected DEFAULT variables, /etc/rabbitmq/rabbitmq.config will be generated when the container starts, and assuming you have the SSL env vars, the configuration should include the ssl related block. You can double check if your config file has been updated by docker-entrypoint.sh by peaking inside the running container with docker exec.

Unfortunately, once SSL was enabled for sure, two other issues emerged when using rabbitmqctl: one of them, you already found out, which is that the variables that configure the Erlang VM to use SSL are not present so you have to add then somehow. The second issue, touches on the hunch you had about permissions, what I found is that if /tmp/combined.pem has root ownership, you get lots of errors on the server regarding SSL, this fix #55 worked for me.

While my goal was not the setup of a secure cluster (at least not now), I still hope it helps you out.

[mancvso]

Thank you very much for your findings! Will try them and inform back.

[yosifkit]

#56 should help when you specify ssl certs and none of default_vhost, default_user, and default_pass.

[mancvso]

Still no luck

EDIT: I just realized it is still not build in the Docker hub

ID=$(docker run -d --name rabbitmq --volumes-from rabbitmq_data -v /root/some1:/etc/cert -e RABBITMQ_DEFAULT_USER=default -e RABBITMQ_DEFAULT_PASS=d3f#ulŧ -e RABBITMQ_ERLANG_COOKIE=#HGDj69A6aNE70MO122FZ -e RABBITMQ_SSL_CERT_FILE=/etc/cert/cert.pem -e RABBITMQ_SSL_KEY_FILE=/etc/cert/key.pem -e RABBITMQ_SSL_CA_FILE=/etc/cert/cacert.pem -e RABBITMQ_NODENAME=rabbit@SOME1 -e TZ=UTC --net=host rabbitmq:3-management)

docker logs $ID

              RabbitMQ 3.6.0. Copyright (C) 2007-2015 Pivotal Software, Inc.
  ##  ##      Licensed under the MPL.  See http://www.rabbitmq.com/
  ##  ##
  ##########  Logs: tty
  ######  ##        tty
  ##########
              Starting broker...
=INFO REPORT==== 6-Jan-2016::19:53:16 ===
Starting RabbitMQ 3.6.0 on Erlang 18.1
Copyright (C) 2007-2015 Pivotal Software, Inc.
Licensed under the MPL.  See http://www.rabbitmq.com/

=INFO REPORT==== 6-Jan-2016::19:53:16 ===
node           : rabbit@SOME1
home dir       : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash    : 8Ga/dCvCNHJupEyUMAsfXQ==
log            : tty
sasl log       : tty
database dir   : /var/lib/rabbitmq/mnesia/rabbit@SOME1

=ERROR REPORT==== 6-Jan-2016::19:53:16 ===
** State machine <0.87.0> terminating 
** Last message in was {'$gen_sync_all_state_event',
                           {<0.29.0>,#Ref<0.0.1.590>},
                           {start,infinity}}
** When State == hello
**      Data  == [{data,
                      [{"StateData",
                        {state,server,
                            {#Ref<0.0.1.591>,<0.29.0>},
                            gen_tcp,tls_connection,tcp,tcp_closed,tcp_error,
                            "localhost",50365,#Port<0.4412>,
                            {ssl_options,tls,
                                [{3,3},{3,2},{3,1}],
                                verify_none,
                                {#Fun<ssl.8.81686783>,[]},
                                #Fun<ssl.9.81686783>,false,false,undefined,1,
                                <<"/tmp/combined.pem">>,"***",
                                <<"/tmp/combined.pem">>,"***","***","***",
                                <<>>,"***",undefined,undefined,"***","***",
                                [<<"�,">>,<<"�0">>,<<"�$">>,<<"�(">>,<<"�.">>,
                                 <<"�2">>,<<"�&">>,<<"�*">>,
                                 <<0,159>>,
                                 <<0,163>>,
                                 <<0,107>>,
                                 <<0,106>>,
                                 <<0,157>>,
                                 <<0,61>>,
                                 <<"�+">>,<<"�/">>,<<"�#">>,<<"�'">>,<<"�-">>,
                                 <<"�1">>,<<"�%">>,<<"�)">>,
                                 <<0,158>>,
                                 <<0,162>>,
                                 <<0,103>>,
                                 <<0,64>>,
                                 <<0,156>>,
                                 <<0,60>>,
                                 <<"�\n">>,
                                 <<192,20>>,
                                 <<0,57>>,
                                 <<0,56>>,
                                 <<192,5>>,
                                 <<192,15>>,
                                 <<0,53>>,
                                 <<"�\b">>,
                                 <<192,18>>,
                                 <<0,22>>,
                                 <<0,19>>,
                                 <<192,3>>,
                                 <<"�\r">>,
                                 <<0,10>>,
                                 <<"�\t">>,
                                 <<192,19>>,
                                 <<0,51>>,
                                 <<0,50>>,
                                 <<192,4>>,
                                 <<192,14>>,
                                 <<0,47>>,
                                 <<0,21>>,
                                 <<0,9>>],
                                #Fun<ssl.2.81686783>,true,268435456,true,true,
                                undefined,true,undefined,undefined,undefined,
                                undefined,true,undefined,[],undefined,false,
                                true,false,false,
                                {ssl_crl_cache,{internal,[]}}},
                            {socket_options,binary,4,0,0,false},
                            "***","***","***",undefined,"***",undefined,
                            ssl_session_cache,undefined,undefined,false,
                            undefined,
                            {undefined,undefined},
                            undefined,undefined,"***","***","***",undefined,
                            "***","***","***",undefined,undefined,undefined,
                            "***",
                            {false,first},
                            undefined,undefined,
                            {[],[]},
                            false,true,false,false,undefined,undefined,
                            undefined,undefined}}]}]
** Reason for termination = 
** {badarg,[{ets,select_delete,
                 [undefined,[{{{undefined,'_','_'},'_'},[],[true]}]],
                 []},
            {ets,match_delete,2,[{file,"ets.erl"},{line,700}]},
            {ssl_pkix_db,remove_certs,2,[{file,"ssl_pkix_db.erl"},{line,243}]},
            {ssl_connection,terminate,3,
                            [{file,"ssl_connection.erl"},{line,941}]},
            {tls_connection,terminate,3,
                            [{file,"tls_connection.erl"},{line,335}]},
            {gen_fsm,terminate,7,[{file,"gen_fsm.erl"},{line,610}]},
            {gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,532}]},
            {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}

=CRASH REPORT==== 6-Jan-2016::19:53:16 ===
  crasher:
    initial call: tls_connection:init/1
    pid: <0.87.0>
    registered_name: []
    exception exit: {badarg,
                        [{ets,select_delete,
                             [undefined,
                              [{{{undefined,'_','_'},'_'},[],[true]}]],
                             []},
                         {ets,match_delete,2,[{file,"ets.erl"},{line,700}]},
                         {ssl_pkix_db,remove_certs,2,
                             [{file,"ssl_pkix_db.erl"},{line,243}]},
                         {ssl_connection,terminate,3,
                             [{file,"ssl_connection.erl"},{line,941}]},
                         {tls_connection,terminate,3,
                             [{file,"tls_connection.erl"},{line,335}]},
                         {gen_fsm,terminate,7,
                             [{file,"gen_fsm.erl"},{line,610}]},
                         {gen_fsm,handle_msg,7,
                             [{file,"gen_fsm.erl"},{line,532}]},
                         {proc_lib,init_p_do_apply,3,
                             [{file,"proc_lib.erl"},{line,240}]}]}
      in function  gen_fsm:handle_msg/7 (gen_fsm.erl, line 535)
    ancestors: [ssl_connection_sup_dist,ssl_dist_sup,net_sup,kernel_sup,
                  <0.10.0>]
    messages: []
    links: [<0.22.0>,#Port<0.4412>]
    dictionary: [{ssl_manager,ssl_managerdist}]
    trap_exit: true
    status: running
    heap_size: 987
    stack_size: 27
    reductions: 328
  neighbours:

=SUPERVISOR REPORT==== 6-Jan-2016::19:53:16 ===
     Supervisor: {local,ssl_connection_sup_dist}
     Context:    child_terminated
     Reason:     {badarg,
                     [{ets,select_delete,
                          [undefined,[{{{undefined,'_','_'},'_'},[],[true]}]],
                          []},
                      {ets,match_delete,2,[{file,"ets.erl"},{line,700}]},
                      {ssl_pkix_db,remove_certs,2,
                          [{file,"ssl_pkix_db.erl"},{line,243}]},
                      {ssl_connection,terminate,3,
                          [{file,"ssl_connection.erl"},{line,941}]},
                      {tls_connection,terminate,3,
                          [{file,"tls_connection.erl"},{line,335}]},
                      {gen_fsm,terminate,7,[{file,"gen_fsm.erl"},{line,610}]},
                      {gen_fsm,handle_msg,7,[{file,"gen_fsm.erl"},{line,532}]},
                      {proc_lib,init_p_do_apply,3,
                          [{file,"proc_lib.erl"},{line,240}]}]}
     Offender:   [{pid,<0.87.0>},
                  {id,undefined},
                  {mfargs,{tls_connection,start_link,undefined}},
                  {restart_type,temporary},
                  {shutdown,4000},
                  {child_type,worker}]

root@some1:/# ls -lh /etc/cert/
total 20K
-rw-r--r-- 1 rabbitmq rabbitmq  712 Dec 11 00:28 cacert.cer
-rw-r--r-- 1 rabbitmq rabbitmq 1021 Jan  6 19:47 cacert.pem
-rw-r--r-- 1 rabbitmq rabbitmq 1.1K Dec 11 00:25 cert.pem
-rw-r--r-- 1 rabbitmq rabbitmq 1.7K Dec 11 00:25 key.pem
-rw-r--r-- 1 rabbitmq rabbitmq  940 Dec 11 00:25 req.pem