Exchange type from plugin does not import from definitions

[caevv]

After manually creating exchanges, I exported it and I'm now trying to import from my rabbitmq.conf.

On the definitions.json the exchanges is part of a installed plugin, but I get this error:

rabbitmq_1  | {"init terminating in do_boot",{error,{error,{amqp_error,command_invalid,"unknown exchange type 'x-message-deduplication'",none}}}}

The plugin is installed as I have tested by manually creating the exchanges without any issues, seems like the definitions is being imported before plugins are enabled somehow? But that would be weird as the plugin is enabled as part of the docker build.

docker-compose.yml

version: '3.7'

services:
    rabbitmq:
      build:
        context: .
      ports:
        - "15672:15672"
        - "5672:5672"
      restart: always
      volumes:
        - ./definitions.json:/opt/rabbitmq/definitions.json
        - ./rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf

Dockerfile

FROM rabbitmq:3.8-management

ADD https://github.com/noxdafox/rabbitmq-message-deduplication/releases/download/0.5.0/elixir-1.10.4.ez /opt/rabbitmq/plugins/
ADD https://github.com/noxdafox/rabbitmq-message-deduplication/releases/download/0.5.0/rabbitmq_message_deduplication-0.5.0.ez /opt/rabbitmq/plugins/

RUN chown rabbitmq:rabbitmq /opt/rabbitmq/plugins/*.ez \
    && rabbitmq-plugins enable --offline rabbitmq_message_deduplication

rabbitmq.conf

loopback_users.guest = false
listeners.tcp.default = 5672
management.tcp.port = 15672
management.load_definitions = /opt/rabbitmq/definitions.json

[wglambert]

I can reproduce, but doing the import manually with rabbitmqctl import_definitions works fine

$ docker build -t rabbitmq:test - << EOF
> FROM rabbitmq:3.8-management
>
> ADD https://github.com/noxdafox/rabbitmq-message-deduplication/releases/download/0.5.0/elixir-1.10.4.ez /opt/rabbitmq/plugins/
> ADD https://github.com/noxdafox/rabbitmq-message-deduplication/releases/download/0.5.0/rabbitmq_message_deduplication-0.5.0.ez /opt/rabbitmq/plugins/
>
> RUN chown rabbitmq:rabbitmq /opt/rabbitmq/plugins/*.ez \
>     && rabbitmq-plugins enable --offline rabbitmq_message_deduplication
> EOF
Sending build context to Docker daemon  2.048kB
Step 1/4 : FROM rabbitmq:3.8-management
 ---> b46cdfa8ade6
Step 2/4 : ADD https://github.com/noxdafox/rabbitmq-message-deduplication/releases/download/0.5.0/elixir-1.10.4.ez /opt/rabbitmq/plugins/
Downloading [==================================================>]  3.303MB/3.303MB
 ---> Using cache
 ---> 9d4d476950a0
Step 3/4 : ADD https://github.com/noxdafox/rabbitmq-message-deduplication/releases/download/0.5.0/rabbitmq_message_deduplication-0.5.0.ez /opt/rabbitmq/plugins/
Downloading [==================================================>]   52.2kB/52.2kB
 ---> Using cache
 ---> 45712fb8dc38
Step 4/4 : RUN chown rabbitmq:rabbitmq /opt/rabbitmq/plugins/*.ez     && rabbitmq-plugins enable --offline rabbitmq_message_deduplication
 ---> Using cache
 ---> fa868687d367
Successfully built fa868687d367
Successfully tagged rabbitmq:test

$ docker run -d --name rabbit -p 15672:15672 -p 5672:5672 rabbitmq:test
e2fee6ffd29f5a174476d4bd940f84b30e117d3bb2b55cb127d85892e0cca292

Create the exchange

Export and copy the definitions out, then re-mount in the docker-compose.yml with the rabbitmq.conf

root@e2fee6ffd29f:~# exit
exit
$ docker cp rabbit:/var/lib/rabbitmq/definitions.json ./

$ chmod 777 ./*

$ ls -al
total 28
drwxrwxrwx  3 batou batou 4096 Dec 28 08:59 .
drwxr-xr-x 24 batou batou 4096 Dec 28 08:59 ..
-rwxrwxrwx  1 batou batou  756 Dec 28 08:56 definitions.json
-rwxrwxrwx  1 batou batou  282 Dec 28 08:59 docker-compose.yml
-rwxrwxrwx  1 batou batou  437 Dec 28 08:30 Dockerfile
-rwxrwxrwx  1 batou batou  147 Dec 28 08:58 rabbitmq.conf

$ docker-compose up -d
Creating network "test_default" with the default driver
Building rabbitmq
Step 1/4 : FROM rabbitmq:3.8-management
 ---> b46cdfa8ade6
Step 2/4 : ADD https://github.com/noxdafox/rabbitmq-message-deduplication/releases/download/0.5.0/elixir-1.10.4.ez /opt/rabbitmq/plugins/

 ---> Using cache
 ---> 9d4d476950a0
Step 3/4 : ADD https://github.com/noxdafox/rabbitmq-message-deduplication/releases/download/0.5.0/rabbitmq_message_deduplication-0.5.0.ez /opt/rabbitmq/plugins/

 ---> Using cache
 ---> 45712fb8dc38
Step 4/4 : RUN chown rabbitmq:rabbitmq /opt/rabbitmq/plugins/*.ez     && rabbitmq-plugins enable --offline rabbitmq_message_deduplication
 ---> Using cache
 ---> fa868687d367

Successfully built fa868687d367
Successfully tagged test_rabbitmq:latest
WARNING: Image for service rabbitmq was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Creating test_rabbitmq_1 ... done 

docker logs snippet of test_rabbitmq_1

. . .
2020-12-28 17:00:06.668 [info] <0.521.0> Successfully set user tags for user 'guest' to [administrator]
2020-12-28 17:00:06.668 [info] <0.526.0> Importing concurrently 1 vhosts...
2020-12-28 17:00:06.668 [info] <0.526.0> Importing concurrently 1 permissions...
2020-12-28 17:00:06.672 [info] <0.521.0> Successfully set permissions for 'guest' in virtual host '/' to '.*', '.*', '.*'
2020-12-28 17:00:06.672 [info] <0.526.0> Importing sequentially 1 global runtime parameters...
2020-12-28 17:00:06.675 [info] <0.526.0> Importing concurrently 1 exchanges...
2020-12-28 17:00:06.675 [error] <0.526.0>

2020-12-28 17:00:06.675 [error] <0.526.0> BOOT FAILED
BOOT FAILED
===========
2020-12-28 17:00:06.675 [error] <0.526.0> ===========
2020-12-28 17:00:06.675 [error] <0.526.0> Exception during startup:
Exception during startup:

    rabbit:do_run_postlaunch_phase/0 line 906
2020-12-28 17:00:06.675 [error] <0.526.0>
2020-12-28 17:00:06.675 [error] <0.526.0>     rabbit:do_run_postlaunch_phase/0 line 906
2020-12-28 17:00:06.675 [error] <0.526.0>     lists:foreach/2 line 1342
    lists:foreach/2 line 1342
2020-12-28 17:00:06.675 [error] <0.526.0>     rabbit:'-do_run_postlaunch_phase/0-fun-12-'/1 line 908
    rabbit:'-do_run_postlaunch_phase/0-fun-12-'/1 line 908
2020-12-28 17:00:06.675 [error] <0.526.0>     rabbit_boot_steps:run_boot_steps/1 line 20
    rabbit_boot_steps:run_boot_steps/1 line 20
2020-12-28 17:00:06.676 [error] <0.526.0>     rabbit_boot_steps:'-run_boot_steps/1-lc$^0/1-0-'/1 line 19
2020-12-28 17:00:06.676 [error] <0.526.0>     rabbit_boot_steps:run_step/2 line 46
    rabbit_boot_steps:'-run_boot_steps/1-lc$^0/1-0-'/1 line 19
    rabbit_boot_steps:run_step/2 line 46
2020-12-28 17:00:06.676 [error] <0.526.0>     rabbit_boot_steps:'-run_step/2-lc$^0/1-0-'/2 line 43
    rabbit_boot_steps:'-run_step/2-lc$^0/1-0-'/2 line 43
2020-12-28 17:00:06.676 [error] <0.526.0> exit:{error,{amqp_error,command_invalid,
exit:{error,{amqp_error,command_invalid,
                        "unknown exchange type 'x-message-deduplication'",
2020-12-28 17:00:06.676 [error] <0.526.0>                         "unknown exchange type 'x-message-deduplication'",
                        none}}
2020-12-28 17:00:06.676 [error] <0.526.0>                         none}}
2020-12-28 17:00:06.676 [error] <0.526.0>

2020-12-28 17:00:07.740 [info] <0.526.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_management
    amqp_client
    rabbitmq_prometheus
    rabbitmq_web_dispatch
    cowboy
    cowlib
    rabbitmq_management_agent
    rabbitmq_message_deduplication
    rabbit
    rabbitmq_prelaunch
    rabbit_common
    sysmon_handler
    os_mon
    ra
    mnesia
2020-12-28 17:00:07.740 [info] <0.526.0> Stopping application 'rabbitmq_management'
2020-12-28 17:00:07.740 [info] <0.526.0> Stopping application 'amqp_client'
2020-12-28 17:00:07.741 [info] <0.44.0> Application amqp_client exited with reason: stopped
2020-12-28 17:00:07.741 [info] <0.526.0> Stopping application 'rabbitmq_prometheus'
2020-12-28 17:00:07.741 [info] <0.44.0> Application amqp_client exited with reason: stopped
2020-12-28 17:00:07.741 [info] <0.526.0> Stopping application 'rabbitmq_web_dispatch'
2020-12-28 17:00:07.743 [info] <0.526.0> Stopping application 'cowboy'
2020-12-28 17:00:07.743 [info] <0.44.0> Application rabbitmq_web_dispatch exited with reason: stopped
2020-12-28 17:00:07.743 [info] <0.44.0> Application rabbitmq_web_dispatch exited with reason: stopped
2020-12-28 17:00:07.744 [info] <0.526.0> Stopping application 'cowlib'
2020-12-28 17:00:07.744 [info] <0.44.0> Application cowboy exited with reason: stopped
2020-12-28 17:00:07.744 [info] <0.44.0> Application cowboy exited with reason: stopped
2020-12-28 17:00:07.744 [info] <0.526.0> Stopping application 'rabbitmq_management_agent'
2020-12-28 17:00:07.744 [info] <0.44.0> Application cowlib exited with reason: stopped
2020-12-28 17:00:07.744 [info] <0.44.0> Application cowlib exited with reason: stopped
2020-12-28 17:00:07.745 [info] <0.526.0> Stopping application 'rabbitmq_message_deduplication'
2020-12-28 17:00:07.745 [info] <0.44.0> Application rabbitmq_management_agent exited with reason: stopped
2020-12-28 17:00:07.745 [info] <0.526.0> Stopping application 'rabbit'
2020-12-28 17:00:07.745 [info] <0.44.0> Application rabbitmq_management_agent exited with reason: stopped
2020-12-28 17:00:07.745 [info] <0.272.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping unregistration.
2020-12-28 17:00:07.746 [info] <0.480.0> Closing all connections in vhost '/' on node 'rabbit@0fcf1e924854' because the vhost is stopping
2020-12-28 17:00:07.746 [info] <0.496.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/rabbit@0fcf1e924854/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent'
2020-12-28 17:00:07.758 [info] <0.496.0> Message store for directory '/var/lib/rabbitmq/mnesia/rabbit@0fcf1e924854/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' is stopped
2020-12-28 17:00:07.758 [info] <0.492.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/rabbit@0fcf1e924854/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient'
2020-12-28 17:00:07.763 [info] <0.492.0> Message store for directory '/var/lib/rabbitmq/mnesia/rabbit@0fcf1e924854/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped
2020-12-28 17:00:07.767 [info] <0.44.0> Application rabbit exited with reason: stopped
2020-12-28 17:00:07.767 [info] <0.44.0> Application rabbit exited with reason: stopped
init terminating in do_boot ({error,{error,{amqp_error,command_invalid,unknown exchange type 'x-message-deduplication',none}}})
{"init terminating in do_boot",{error,{error,{amqp_error,command_invalid,"unknown exchange type 'x-message-deduplication'",none}}}}

Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done

---

Manually importing the definitions.json:

$ docker run -d --name rabbit -p 15672:15672 -p 5672:5672 -v "$PWD"/definitions.json:/opt/rabbitmq/definitions.json rabbitmq:test
d4ce28a4f890c5a6aa67e4a1be3964677d1d0e0ec1cf00c1bb7362be198396fb

$ docker exec -it rabbit bash

root@d4ce28a4f890:/# rabbitmqctl import_definitions /opt/rabbitmq/definitions.json
Importing definitions in JSON from a file at "/opt/rabbitmq/definitions.json" ...
Successfully started definition import. This process is asynchronous and can take some time.

root@d4ce28a4f890:/# rabbitmqctl list_exchanges
Listing exchanges for vhost / ...
name    type
amq.topic       topic
        direct
amq.fanout      fanout
amq.rabbitmq.trace      topic
amq.match       headers
amq.headers     headers
amq.direct      direct
test    x-message-deduplication

[michaelklishin]

This is a very old chicken-and-egg problem exacerbated by internal refactoring in 3.8.4 and addressed starting with RabbitMQ 3.8.6 rabbitmq/rabbitmq-server#2384. It has nothing to do with this image.

In order to benefit from the above changes, replace management.load_definitions with load_definitions as management plugin cannot guarantee that it will be activated after your custom exchange type plugins. This image already does this as of #430.

[michaelklishin]

@wglambert this is a duplicate of #429. Please close.