No rabbitmq logs {removed_failing_handler,rabbit_log}

[jaschaio]

I don't get any rabbitmq logs beside the start up ones. Instead I see {removed_failing_handler,rabbit_log} when I would expect the first log to appear (e.G. an application making a connection):

touch: cannot touch '/etc/rabbitmq/rabbitmq.conf': Read-only file system

WARNING: '/etc/rabbitmq/rabbitmq.conf' is not writable, but environment variables have been provided which request that we write to it
  We have copied it to '/tmp/rabbitmq.conf' so it can be amended to work around the problem, but it is recommended that the read-only source file should be modified and the environment variables removed instead.

Configuring logger redirection
2021-03-21 18:50:35.161 [debug] <0.294.0> Lager installed handler error_logger_lager_h into error_logger
2021-03-21 18:50:35.174 [debug] <0.297.0> Lager installed handler lager_forwarder_backend into error_logger_lager_event
2021-03-21 18:50:35.174 [debug] <0.300.0> Lager installed handler lager_forwarder_backend into rabbit_log_lager_event
2021-03-21 18:50:35.176 [debug] <0.303.0> Lager installed handler lager_forwarder_backend into rabbit_log_channel_lager_event
2021-03-21 18:50:35.180 [debug] <0.306.0> Lager installed handler lager_forwarder_backend into rabbit_log_connection_lager_event
2021-03-21 18:50:35.188 [debug] <0.309.0> Lager installed handler lager_forwarder_backend into rabbit_log_feature_flags_lager_event
2021-03-21 18:50:35.189 [debug] <0.312.0> Lager installed handler lager_forwarder_backend into rabbit_log_federation_lager_event
2021-03-21 18:50:35.191 [debug] <0.315.0> Lager installed handler lager_forwarder_backend into rabbit_log_ldap_lager_event
2021-03-21 18:50:35.194 [debug] <0.318.0> Lager installed handler lager_forwarder_backend into rabbit_log_mirroring_lager_event
2021-03-21 18:50:35.199 [debug] <0.321.0> Lager installed handler lager_forwarder_backend into rabbit_log_prelaunch_lager_event
2021-03-21 18:50:35.202 [debug] <0.324.0> Lager installed handler lager_forwarder_backend into rabbit_log_queue_lager_event
2021-03-21 18:50:35.206 [debug] <0.327.0> Lager installed handler lager_forwarder_backend into rabbit_log_ra_lager_event
2021-03-21 18:50:35.208 [debug] <0.330.0> Lager installed handler lager_forwarder_backend into rabbit_log_shovel_lager_event
2021-03-21 18:50:35.212 [debug] <0.333.0> Lager installed handler lager_forwarder_backend into rabbit_log_upgrade_lager_event
2021-03-21 18:50:35.268 [info] <0.44.0> Application lager started on node 'rabbit@rabbit-01'
2021-03-21 18:50:35.659 [debug] <0.290.0> Lager installed handler lager_backend_throttle into lager_event
2021-03-21 18:50:38.714 [info] <0.44.0> Application mnesia started on node 'rabbit@rabbit-01'

  ##  ##      RabbitMQ 3.8.14
  ##  ##
  ##########  Copyright (c) 2007-2021 VMware, Inc. or its affiliates.
  ######  ##
  ##########  Licensed under the MPL 2.0. Website: https://rabbitmq.com

  Doc guides: https://rabbitmq.com/documentation.html
  Support:    https://rabbitmq.com/contact.html
  Tutorials:  https://rabbitmq.com/getstarted.html
  Monitoring: https://rabbitmq.com/monitoring.html

  Logs: <stdout>

  Config file(s): /tmp/rabbitmq.conf

  Starting broker...2021-03-21 18:50:38.715 [info] <0.279.0>
 Starting RabbitMQ 3.8.14 on Erlang 23.2.7
 Copyright (c) 2007-2021 VMware, Inc. or its affiliates.
 Licensed under the MPL 2.0. Website: https://rabbitmq.com
2021-03-21 18:50:38.717 [info] <0.279.0>
 node           : rabbit@rabbit-01
 home dir       : /var/lib/rabbitmq
 config file(s) : /tmp/rabbitmq.conf
 cookie hash    : Wshq4WD2kDufnsGLpnfzYnA==
 log(s)         : <stdout>
 database dir   : /var/lib/rabbitmq/mnesia/rabbit@rabbit-01
2021-03-21 18:50:39.821 [info] <0.279.0> Feature flags: list of feature flags found:
2021-03-21 18:50:39.822 [info] <0.279.0> Feature flags:   [x] drop_unroutable_metric
2021-03-21 18:50:39.822 [info] <0.279.0> Feature flags:   [x] empty_basic_get_metric
2021-03-21 18:50:39.822 [info] <0.279.0> Feature flags:   [x] implicit_default_bindings
2021-03-21 18:50:39.822 [info] <0.279.0> Feature flags:   [x] maintenance_mode_status
2021-03-21 18:50:39.822 [info] <0.279.0> Feature flags:   [x] quorum_queue
2021-03-21 18:50:39.822 [info] <0.279.0> Feature flags:   [x] user_limits
2021-03-21 18:50:39.822 [info] <0.279.0> Feature flags:   [x] virtual_host_metadata
2021-03-21 18:50:39.822 [info] <0.279.0> Feature flags: feature flag states written to disk: yes
2021-03-21 18:50:40.258 [info] <0.279.0> Running boot step pre_boot defined by app rabbit
2021-03-21 18:50:40.258 [info] <0.279.0> Running boot step rabbit_core_metrics defined by app rabbit
2021-03-21 18:50:40.259 [info] <0.279.0> Running boot step rabbit_alarm defined by app rabbit
2021-03-21 18:50:40.263 [info] <0.448.0> Memory high watermark set to 3114 MiB (3265884979 bytes) of 7786 MiB (8164712448 bytes) total
2021-03-21 18:50:40.268 [info] <0.450.0> Enabling free disk space monitoring
2021-03-21 18:50:40.268 [info] <0.450.0> Disk free limit set to 50MB
2021-03-21 18:50:40.273 [info] <0.279.0> Running boot step code_server_cache defined by app rabbit
2021-03-21 18:50:40.273 [info] <0.279.0> Running boot step file_handle_cache defined by app rabbit
2021-03-21 18:50:40.273 [info] <0.453.0> Limiting to approx 1048479 file handles (943629 sockets)
2021-03-21 18:50:40.274 [info] <0.454.0> FHC read buffering:  OFF
2021-03-21 18:50:40.274 [info] <0.454.0> FHC write buffering: ON
2021-03-21 18:50:40.274 [info] <0.279.0> Running boot step worker_pool defined by app rabbit
2021-03-21 18:50:40.275 [info] <0.395.0> Will use 2 processes for default worker pool
2021-03-21 18:50:40.275 [info] <0.395.0> Starting worker pool 'worker_pool' with 2 processes in it
2021-03-21 18:50:40.275 [info] <0.279.0> Running boot step database defined by app rabbit
2021-03-21 18:50:41.508 [info] <0.279.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-03-21 18:50:41.508 [info] <0.279.0> Successfully synced tables from a peer
2021-03-21 18:50:41.508 [info] <0.279.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-03-21 18:50:41.509 [info] <0.279.0> Successfully synced tables from a peer
2021-03-21 18:50:41.569 [info] <0.279.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-03-21 18:50:41.569 [info] <0.279.0> Successfully synced tables from a peer
2021-03-21 18:50:41.569 [info] <0.279.0> Will register with peer discovery backend rabbit_peer_discovery_classic_config
2021-03-21 18:50:41.569 [info] <0.279.0> Running boot step database_sync defined by app rabbit
2021-03-21 18:50:41.569 [info] <0.279.0> Running boot step feature_flags defined by app rabbit
2021-03-21 18:50:41.569 [info] <0.279.0> Running boot step codec_correctness_check defined by app rabbit
2021-03-21 18:50:41.570 [info] <0.279.0> Running boot step external_infrastructure defined by app rabbit
2021-03-21 18:50:41.570 [info] <0.279.0> Running boot step rabbit_registry defined by app rabbit
2021-03-21 18:50:41.570 [info] <0.279.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2021-03-21 18:50:41.570 [info] <0.279.0> Running boot step rabbit_queue_location_random defined by app rabbit
2021-03-21 18:50:41.570 [info] <0.279.0> Running boot step rabbit_event defined by app rabbit
2021-03-21 18:50:41.570 [info] <0.279.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2021-03-21 18:50:41.570 [info] <0.279.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2021-03-21 18:50:41.570 [info] <0.279.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2021-03-21 18:50:41.571 [info] <0.279.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2021-03-21 18:50:41.571 [info] <0.279.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2021-03-21 18:50:41.571 [info] <0.279.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2021-03-21 18:50:41.571 [info] <0.279.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2021-03-21 18:50:41.571 [info] <0.279.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2021-03-21 18:50:41.571 [info] <0.279.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2021-03-21 18:50:41.571 [info] <0.279.0> Running boot step rabbit_priority_queue defined by app rabbit
2021-03-21 18:50:41.571 [info] <0.279.0> Priority queues enabled, real BQ is rabbit_variable_queue
2021-03-21 18:50:41.571 [info] <0.279.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2021-03-21 18:50:41.571 [info] <0.279.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2021-03-21 18:50:41.571 [info] <0.279.0> Running boot step kernel_ready defined by app rabbit
2021-03-21 18:50:41.571 [info] <0.279.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2021-03-21 18:50:41.572 [info] <0.279.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2021-03-21 18:50:41.572 [info] <0.483.0> epmd monitor knows us, inter-node communication (distribution) port: 25672
2021-03-21 18:50:41.572 [info] <0.279.0> Running boot step guid_generator defined by app rabbit
2021-03-21 18:50:41.589 [info] <0.279.0> Running boot step rabbit_node_monitor defined by app rabbit
2021-03-21 18:50:41.590 [info] <0.487.0> Starting rabbit_node_monitor
2021-03-21 18:50:41.591 [info] <0.279.0> Running boot step delegate_sup defined by app rabbit
2021-03-21 18:50:41.594 [info] <0.279.0> Running boot step rabbit_memory_monitor defined by app rabbit
2021-03-21 18:50:41.595 [info] <0.279.0> Running boot step core_initialized defined by app rabbit
2021-03-21 18:50:41.595 [info] <0.279.0> Running boot step upgrade_queues defined by app rabbit
2021-03-21 18:50:41.636 [info] <0.279.0> Running boot step channel_tracking defined by app rabbit
2021-03-21 18:50:41.637 [info] <0.279.0> Setting up a table for channel tracking on this node: 'tracked_channel_on_node_rabbit@rabbit-01'
2021-03-21 18:50:41.637 [info] <0.279.0> Setting up a table for channel tracking on this node: 'tracked_channel_table_per_user_on_node_rabbit@rabbit-01'
2021-03-21 18:50:41.638 [info] <0.279.0> Running boot step rabbit_channel_tracking_handler defined by app rabbit
2021-03-21 18:50:41.638 [info] <0.279.0> Running boot step connection_tracking defined by app rabbit
2021-03-21 18:50:41.639 [info] <0.279.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_rabbit@rabbit-01'
2021-03-21 18:50:41.639 [info] <0.279.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_rabbit@rabbit-01'
2021-03-21 18:50:41.639 [info] <0.279.0> Setting up a table for per-user connection counting on this node: 'tracked_connection_table_per_user_on_node_rabbit@rabbit-01'
2021-03-21 18:50:41.640 [info] <0.279.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2021-03-21 18:50:41.640 [info] <0.279.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2021-03-21 18:50:41.640 [info] <0.279.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2021-03-21 18:50:41.641 [info] <0.279.0> Running boot step rabbit_policies defined by app rabbit
2021-03-21 18:50:41.644 [info] <0.279.0> Running boot step rabbit_policy defined by app rabbit
2021-03-21 18:50:41.645 [info] <0.279.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2021-03-21 18:50:41.645 [info] <0.279.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit
2021-03-21 18:50:41.645 [info] <0.279.0> Running boot step rabbit_vhost_limit defined by app rabbit
2021-03-21 18:50:41.645 [info] <0.279.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2021-03-21 18:50:41.645 [info] <0.279.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2021-03-21 18:50:41.645 [info] <0.279.0> Management plugin: using rates mode 'basic'
2021-03-21 18:50:41.647 [info] <0.279.0> Running boot step recovery defined by app rabbit
2021-03-21 18:50:41.649 [info] <0.523.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@rabbit-01/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2021-03-21 18:50:41.662 [info] <0.523.0> Starting message stores for vhost '/'
2021-03-21 18:50:41.663 [info] <0.527.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2021-03-21 18:50:41.669 [info] <0.523.0> Started message store of type transient for vhost '/'
2021-03-21 18:50:41.670 [info] <0.531.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2021-03-21 18:50:41.676 [info] <0.523.0> Started message store of type persistent for vhost '/'
2021-03-21 18:50:41.726 [info] <0.279.0> Running boot step empty_db_check defined by app rabbit
2021-03-21 18:50:41.726 [info] <0.279.0> Will not seed default virtual host and user: have definitions to load...
2021-03-21 18:50:41.726 [info] <0.279.0> Running boot step rabbit_looking_glass defined by app rabbit
2021-03-21 18:50:41.726 [info] <0.279.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2021-03-21 18:50:41.727 [info] <0.279.0> Running boot step background_gc defined by app rabbit
2021-03-21 18:50:41.727 [info] <0.279.0> Running boot step routing_ready defined by app rabbit
2021-03-21 18:50:41.728 [info] <0.279.0> Running boot step pre_flight defined by app rabbit
2021-03-21 18:50:41.728 [info] <0.279.0> Running boot step notify_cluster defined by app rabbit
2021-03-21 18:50:41.728 [info] <0.279.0> Running boot step networking defined by app rabbit
2021-03-21 18:50:41.728 [info] <0.279.0> Running boot step definition_import_worker_pool defined by app rabbit
2021-03-21 18:50:41.728 [info] <0.395.0> Starting worker pool 'definition_import_pool' with 2 processes in it
2021-03-21 18:50:41.730 [info] <0.279.0> Running boot step cluster_name defined by app rabbit
2021-03-21 18:50:41.731 [info] <0.279.0> Running boot step direct_client defined by app rabbit
2021-03-21 18:50:41.731 [info] <0.279.0> Running boot step rabbit_management_load_definitions defined by app rabbitmq_management
2021-03-21 18:50:41.731 [info] <0.564.0> Resetting node maintenance status
2021-03-21 18:50:41.732 [info] <0.44.0> Application rabbit started on node 'rabbit@rabbit-01'
2021-03-21 18:50:41.738 [info] <0.44.0> Application rabbitmq_management_agent started on node 'rabbit@rabbit-01'
2021-03-21 18:50:41.739 [info] <0.44.0> Application cowlib started on node 'rabbit@rabbit-01'
2021-03-21 18:50:41.740 [info] <0.44.0> Application cowboy started on node 'rabbit@rabbit-01'
2021-03-21 18:50:41.741 [info] <0.44.0> Application rabbitmq_web_dispatch started on node 'rabbit@rabbit-01'
2021-03-21 18:50:41.741 [info] <0.44.0> Application amqp_client started on node 'rabbit@rabbit-01'
2021-03-21 18:50:41.787 [info] <0.625.0> Management plugin: HTTP (non-TLS) listener started on port 15672
2021-03-21 18:50:41.829 [info] <0.625.0> Management plugin: HTTPS listener started on port 15671
2021-03-21 18:50:41.831 [info] <0.836.0> Statistics database started.
2021-03-21 18:50:41.831 [info] <0.835.0> Starting worker pool 'management_worker_pool' with 3 processes in it
2021-03-21 18:50:41.834 [info] <0.44.0> Application rabbitmq_management started on node 'rabbit@rabbit-01'
2021-03-21 18:50:41.877 [info] <0.44.0> Application prometheus started on node 'rabbit@rabbit-01'
2021-03-21 18:50:41.881 [info] <0.849.0> Prometheus metrics: HTTP (non-TLS) listener started on port 15692
2021-03-21 18:50:41.882 [info] <0.564.0> Ready to start client connection listeners
2021-03-21 18:50:41.882 [info] <0.44.0> Application rabbitmq_prometheus started on node 'rabbit@rabbit-01'
2021-03-21 18:50:41.887 [info] <0.969.0> started TLS (SSL) listener on [::]:5671
2021-03-21 18:50:42.638 [info] <0.564.0> Server startup complete; 4 plugins started.
 * rabbitmq_prometheus
 * rabbitmq_management
 * rabbitmq_web_dispatch
 * rabbitmq_management_agent
 completed with 4 plugins.
2021-03-21 18:50:42.639 [info] <0.564.0> Resetting node maintenance status
{removed_failing_handler,rabbit_log}

This is a three node cluster. In my docker compose file I am just setting the SSL environment variables and user and password:

            RABBITMQ_MANAGEMENT_SSL_CACERTFILE: /run/secrets/ca_cert.pem
            RABBITMQ_MANAGEMENT_SSL_CERTFILE: /run/secrets/cert.pem
            RABBITMQ_MANAGEMENT_SSL_KEYFILE: /run/secrets/privkey.pem
            RABBITMQ_MANAGEMENT_SSL_VERIFY: verify_peer
            RABBITMQ_MANAGEMENT_SSL_FAIL_IF_NO_PEER_CERT: "true"
            RABBITMQ_SSL_CACERTFILE: /run/secrets/ca_cert.pem
            RABBITMQ_SSL_CERTFILE: /run/secrets/cert.pem
            RABBITMQ_SSL_KEYFILE: /run/secrets/privkey.pem
            RABBITMQ_SSL_VERIFY: verify_peer
            RABBITMQ_SSL_FAIL_IF_NO_PEER_CERT: "true"
            RABBITMQ_DEFAULT_USER_FILE: /run/secrets/rabbit_user
            RABBITMQ_DEFAULT_PASS_FILE: /run/secrets/rabbit_pass
            RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS: -pa $ERL_SSL_PATH -proto_dist inet_tls -ssl_dist_opt server_certfile /run/secrets/fullchain.pem

My config file is fairly simple:

listeners.tcp = none

ssl_options.versions.1 = tlsv1.3
ssl_options.ciphers.1  = TLS_AES_256_GCM_SHA384
ssl_options.ciphers.2  = TLS_AES_128_GCM_SHA256
ssl_options.ciphers.3  = TLS_CHACHA20_POLY1305_SHA256
ssl_options.ciphers.4  = TLS_AES_128_CCM_SHA256
ssl_options.ciphers.5  = TLS_AES_128_CCM_8_SHA256

cluster_formation.peer_discovery_backend = classic_config

cluster_formation.classic_config.nodes.1 = rabbit@rabbit-01
cluster_formation.classic_config.nodes.2 = rabbit@rabbit-02
cluster_formation.classic_config.nodes.3 = rabbit@rabbit-03

loopback_users.guest = true

RabbitMQ Version rabbitmq:3.8.14-management

[wglambert]

Are you mounting the rabbitmq.conf? Looks like it needs its permissions changed to fit the user in the container

[jaschaio]

Mounting it via docker config, which is always read-only no matter which permissions or user its mounted to.

But I don't think that's the issue as I don't have anything logging wise in the config itself.

[yosifkit]

log(s) : <stdout>

The image defaults to log to stdout; not sure how that would fail.

---

Not that this will fix the log handler failing, but if you are already setting up a config file for rabbitmq, why even use the environment variables that we then have to set into a new config file (since the provided config is read-only). You should probably just put your config in the file and skip the variables (#424).

[jaschaio]

Ok, I am mainly using the environment variables to set the default_user and default_pass from secrets via RABBITMQ_DEFAULT_USER_FILE and RABBITMQ_DEFAULT_PASS_FILE.

But let's say I use a docker secret instead of a docker config, include the default_user and default_pass directly within it and mount that into the container.

Then I still get the {removed_failing_handler,rabbit_log} error (see example logs below).

As far as I can see the config file is exactly the same in both cases, the only difference is that it's now located at /etc/rabbitmq/rabbitmq.conf instead of /tmp/rabbitmq.conf. So this doesn't seem to be caused by using environment variables vs. a config. It doesn't seem either that stdout is failing, as first there is some output – but that the Logging Handler fails at some point for an unclear reason.

rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | Configuring logger redirection
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 11:33:33.185 [warning] segment_writer: failed to open segment file /var/lib/rabbitmq/mnesia/rabbit@rabbit-01/quorum/rabbit@rabbit-01/2F_TESGGQTSJTL64IL/00000001.segmenterror: enoent
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 11:33:33.209 [warning] segment_writer: skipping segment as directory /var/lib/rabbitmq/mnesia/rabbit@rabbit-01/quorum/rabbit@rabbit-01/2F_TESGGQTSJTL64IL does not exist
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:34.658 [debug] <0.325.0> Lager installed handler error_logger_lager_h into error_logger
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:34.658 [debug] <0.328.0> Lager installed handler lager_forwarder_backend into error_logger_lager_event
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:34.661 [debug] <0.331.0> Lager installed handler lager_forwarder_backend into rabbit_log_lager_event
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:34.663 [debug] <0.334.0> Lager installed handler lager_forwarder_backend into rabbit_log_channel_lager_event
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:34.667 [debug] <0.337.0> Lager installed handler lager_forwarder_backend into rabbit_log_connection_lager_event
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:34.669 [debug] <0.340.0> Lager installed handler lager_forwarder_backend into rabbit_log_feature_flags_lager_event
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:34.672 [debug] <0.343.0> Lager installed handler lager_forwarder_backend into rabbit_log_federation_lager_event
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:34.675 [debug] <0.346.0> Lager installed handler lager_forwarder_backend into rabbit_log_ldap_lager_event
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:34.677 [debug] <0.349.0> Lager installed handler lager_forwarder_backend into rabbit_log_mirroring_lager_event
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:34.680 [debug] <0.352.0> Lager installed handler lager_forwarder_backend into rabbit_log_prelaunch_lager_event
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:34.684 [debug] <0.355.0> Lager installed handler lager_forwarder_backend into rabbit_log_queue_lager_event
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:34.687 [debug] <0.358.0> Lager installed handler lager_forwarder_backend into rabbit_log_ra_lager_event
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:34.688 [debug] <0.361.0> Lager installed handler lager_forwarder_backend into rabbit_log_shovel_lager_event
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:34.691 [debug] <0.364.0> Lager installed handler lager_forwarder_backend into rabbit_log_upgrade_lager_event
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:34.733 [info] <0.44.0> Application lager started on node 'rabbit@rabbit-01'
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:35.156 [debug] <0.321.0> Lager installed handler lager_backend_throttle into lager_event
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:38.737 [info] <0.44.0> Application mnesia started on node 'rabbit@rabbit-01'
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:38.739 [info] <0.303.0>
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |  Starting RabbitMQ 3.8.14 on Erlang 23.2.7
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |  Copyright (c) 2007-2021 VMware, Inc. or its affiliates.
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |  Licensed under the MPL 2.0. Website: https://rabbitmq.com
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |   ##  ##      RabbitMQ 3.8.14
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |   ##  ##
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |   ##########  Copyright (c) 2007-2021 VMware, Inc. or its affiliates.
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |   ######  ##
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |   ##########  Licensed under the MPL 2.0. Website: https://rabbitmq.com
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |   Doc guides: https://rabbitmq.com/documentation.html
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |   Support:    https://rabbitmq.com/contact.html
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |   Tutorials:  https://rabbitmq.com/getstarted.html
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |   Monitoring: https://rabbitmq.com/monitoring.html
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |   Logs: <stdout>
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |   Config file(s): /etc/rabbitmq/rabbitmq.conf
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |   Starting broker...2021-03-23 11:33:38.742 [info] <0.303.0>
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |  node           : rabbit@rabbit-01
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |  home dir       : /var/lib/rabbitmq
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |  config file(s) : /etc/rabbitmq/rabbitmq.conf
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |  cookie hash    : Wshq4WD2GY7JmGLpnfzYnA==
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |  log(s)         : <stdout>
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |  database dir   : /var/lib/rabbitmq/mnesia/rabbit@rabbit-01
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:39.788 [info] <0.303.0> Feature flags: list of feature flags found:
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:39.788 [info] <0.303.0> Feature flags:   [x] drop_unroutable_metric
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:39.789 [info] <0.303.0> Feature flags:   [x] empty_basic_get_metric
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:39.789 [info] <0.303.0> Feature flags:   [x] implicit_default_bindings
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:39.789 [info] <0.303.0> Feature flags:   [x] maintenance_mode_status
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:39.790 [info] <0.303.0> Feature flags:   [x] quorum_queue
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:39.790 [info] <0.303.0> Feature flags:   [x] user_limits
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:39.790 [info] <0.303.0> Feature flags:   [x] virtual_host_metadata
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:39.791 [info] <0.303.0> Feature flags: feature flag states written to disk: yes
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.198 [info] <0.303.0> Running boot step pre_boot defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.199 [info] <0.303.0> Running boot step rabbit_core_metrics defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.200 [info] <0.303.0> Running boot step rabbit_alarm defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.211 [info] <0.508.0> Memory high watermark set to 3114 MiB (3265884979 bytes) of 7786 MiB (8164712448 bytes) total
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.223 [info] <0.510.0> Enabling free disk space monitoring
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.223 [info] <0.510.0> Disk free limit set to 50MB
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.230 [info] <0.303.0> Running boot step code_server_cache defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.230 [info] <0.303.0> Running boot step file_handle_cache defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.231 [info] <0.513.0> Limiting to approx 1048479 file handles (943629 sockets)
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.231 [info] <0.514.0> FHC read buffering:  OFF
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.231 [info] <0.514.0> FHC write buffering: ON
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.232 [info] <0.303.0> Running boot step worker_pool defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.232 [info] <0.425.0> Will use 2 processes for default worker pool
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.233 [info] <0.425.0> Starting worker pool 'worker_pool' with 2 processes in it
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.233 [info] <0.303.0> Running boot step database defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.352 [info] <0.303.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.353 [info] <0.303.0> Successfully synced tables from a peer
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.355 [info] <0.303.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.355 [info] <0.303.0> Successfully synced tables from a peer
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.414 [info] <0.303.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.415 [info] <0.303.0> Successfully synced tables from a peer
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.475 [info] <0.303.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.475 [info] <0.303.0> Successfully synced tables from a peer
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.475 [info] <0.303.0> Will register with peer discovery backend rabbit_peer_discovery_classic_config
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.476 [info] <0.303.0> Running boot step database_sync defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.476 [info] <0.303.0> Running boot step feature_flags defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.477 [info] <0.303.0> Running boot step codec_correctness_check defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.477 [info] <0.303.0> Running boot step external_infrastructure defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.477 [info] <0.303.0> Running boot step rabbit_registry defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.477 [info] <0.303.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.477 [info] <0.303.0> Running boot step rabbit_queue_location_random defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.477 [info] <0.303.0> Running boot step rabbit_event defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.477 [info] <0.303.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.477 [info] <0.303.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.477 [info] <0.303.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.477 [info] <0.303.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.478 [info] <0.303.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.478 [info] <0.303.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.478 [info] <0.303.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.478 [info] <0.303.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.478 [info] <0.303.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.478 [info] <0.303.0> Running boot step rabbit_priority_queue defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.478 [info] <0.303.0> Priority queues enabled, real BQ is rabbit_variable_queue
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.478 [info] <0.303.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.478 [info] <0.303.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.479 [info] <0.303.0> Running boot step kernel_ready defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.479 [info] <0.303.0> Running boot step rabbit_sysmon_minder defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.479 [info] <0.303.0> Running boot step rabbit_epmd_monitor defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.480 [info] <0.536.0> epmd monitor knows us, inter-node communication (distribution) port: 25672
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.483 [info] <0.303.0> Running boot step guid_generator defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.492 [info] <0.303.0> Running boot step rabbit_node_monitor defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.524 [info] <0.540.0> Starting rabbit_node_monitor
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.525 [info] <0.303.0> Running boot step delegate_sup defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.528 [info] <0.303.0> Running boot step rabbit_memory_monitor defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.528 [info] <0.303.0> Running boot step core_initialized defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.529 [info] <0.303.0> Running boot step upgrade_queues defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.567 [info] <0.303.0> Running boot step channel_tracking defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.622 [info] <0.303.0> Setting up a table for channel tracking on this node: 'tracked_channel_on_node_rabbit@rabbit-01'
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.649 [info] <0.303.0> Setting up a table for channel tracking on this node: 'tracked_channel_table_per_user_on_node_rabbit@rabbit-01'
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.649 [info] <0.303.0> Running boot step rabbit_channel_tracking_handler defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.650 [info] <0.303.0> Running boot step connection_tracking defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.677 [info] <0.303.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_rabbit@rabbit-01'
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.708 [info] <0.303.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_rabbit@rabbit-01'
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.740 [info] <0.303.0> Setting up a table for per-user connection counting on this node: 'tracked_connection_table_per_user_on_node_rabbit@rabbit-01'
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.740 [info] <0.303.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.740 [info] <0.303.0> Running boot step rabbit_exchange_parameters defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.741 [info] <0.303.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.741 [info] <0.303.0> Running boot step rabbit_policies defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.742 [info] <0.303.0> Running boot step rabbit_policy defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.743 [info] <0.303.0> Running boot step rabbit_queue_location_validator defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.743 [info] <0.303.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.743 [info] <0.303.0> Running boot step rabbit_vhost_limit defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.743 [info] <0.303.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.743 [info] <0.303.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.743 [info] <0.303.0> Management plugin: using rates mode 'basic'
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.825 [info] <0.303.0> Running boot step recovery defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.826 [error] <0.303.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2278823811.3125805057.102929>}} from <0.303.0> to <0.18025.0> in an old incarnation (1616412295) of this node (1616499213)
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.826 [error] <0.303.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2278823811.3125805057.102929>}} from <0.303.0> to <0.18534.0> in an old incarnation (1616412295) of this node (1616499213)
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.826 [error] emulator Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2278823811.3125805057.102929>}} from <0.303.0> to <0.18025.0> in an old incarnation (1616412295) of this node (1616499213)
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.827 [error] <0.303.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2278823811.3125805057.102929>}} from <0.303.0> to <0.17710.0> in an old incarnation (1616412295) of this node (1616499213)
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.827 [error] <0.303.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2278823811.3125805057.102929>}} from <0.303.0> to <0.1956.0> in an old incarnation (1616412295) of this node (1616499213)
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.827 [error] emulator Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2278823811.3125805057.102929>}} from <0.303.0> to <0.18534.0> in an old incarnation (1616412295) of this node (1616499213)
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.827 [error] <0.303.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2278823811.3125805057.102929>}} from <0.303.0> to <0.17216.0> in an old incarnation (1616412295) of this node (1616499213)
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.827 [error] <0.303.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2278823811.3125805057.102929>}} from <0.303.0> to <0.17875.0> in an old incarnation (1616412295) of this node (1616499213)
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.828 [error] <0.303.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2278823811.3125805057.102929>}} from <0.303.0> to <0.17551.0> in an old incarnation (1616412295) of this node (1616499213)
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.827 [error] emulator Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2278823811.3125805057.102929>}} from <0.303.0> to <0.17710.0> in an old incarnation (1616412295) of this node (1616499213)
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.828 [error] <0.303.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2278823811.3125805057.102929>}} from <0.303.0> to <0.18686.0> in an old incarnation (1616412295) of this node (1616499213)
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.828 [error] <0.303.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2278823811.3125805057.102929>}} from <0.303.0> to <0.18357.0> in an old incarnation (1616412295) of this node (1616499213)
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.828 [error] <0.303.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2278823811.3125805057.102929>}} from <0.303.0> to <0.1889.0> in an old incarnation (1616412295) of this node (1616499213)
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.828 [error] emulator Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2278823811.3125805057.102929>}} from <0.303.0> to <0.1956.0> in an old incarnation (1616412295) of this node (1616499213)
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.828 [error] <0.303.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2278823811.3125805057.102929>}} from <0.303.0> to <0.18218.0> in an old incarnation (1616412295) of this node (1616499213)
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.828 [error] <0.303.0> Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2278823811.3125805057.102929>}} from <0.303.0> to <0.18880.0> in an old incarnation (1616412295) of this node (1616499213)
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.828 [error] emulator Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2278823811.3125805057.102929>}} from <0.303.0> to <0.17216.0> in an old incarnation (1616412295) of this node (1616499213)
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.829 [error] emulator Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2278823811.3125805057.102929>}} from <0.303.0> to <0.17875.0> in an old incarnation (1616412295) of this node (1616499213)
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.829 [error] emulator Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2278823811.3125805057.102929>}} from <0.303.0> to <0.17551.0> in an old incarnation (1616412295) of this node (1616499213)
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.829 [error] emulator Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2278823811.3125805057.102929>}} from <0.303.0> to <0.18686.0> in an old incarnation (1616412295) of this node (1616499213)
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.829 [error] emulator Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2278823811.3125805057.102929>}} from <0.303.0> to <0.18357.0> in an old incarnation (1616412295) of this node (1616499213)
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.829 [error] emulator Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2278823811.3125805057.102929>}} from <0.303.0> to <0.1889.0> in an old incarnation (1616412295) of this node (1616499213)
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.829 [error] emulator Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2278823811.3125805057.102929>}} from <0.303.0> to <0.18218.0> in an old incarnation (1616412295) of this node (1616499213)
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:40.829 [error] emulator Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.2278823811.3125805057.102929>}} from <0.303.0> to <0.18880.0> in an old incarnation (1616412295) of this node (1616499213)
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:42.056 [info] <0.606.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@rabbit-01/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:42.071 [info] <0.606.0> Starting message stores for vhost '/'
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:42.071 [info] <0.610.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:42.075 [info] <0.606.0> Started message store of type transient for vhost '/'
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:42.076 [info] <0.614.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:42.082 [info] <0.606.0> Started message store of type persistent for vhost '/'
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:43.575 [warning] <0.629.0> Mirrored queue 'retry-05-32000ms' in vhost '/': Unable to start queue mirror on node ''rabbit@rabbit-02''. Target virtual host is not running: {vhost_supervisor_not_running,<<"/">>}
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:43.581 [warning] <0.623.0> Mirrored queue 'retry-07-128000ms' in vhost '/': Unable to start queue mirror on node ''rabbit@rabbit-02''. Target virtual host is not running: {vhost_supervisor_not_running,<<"/">>}
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:43.691 [warning] <0.632.0> Mirrored queue 'retry-02-4000ms' in vhost '/': Unable to start queue mirror on node ''rabbit@rabbit-02''. Target virtual host is not running: {vhost_supervisor_not_running,<<"/">>}
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:43.696 [warning] <0.635.0> Mirrored queue 'retry-03-8000ms' in vhost '/': Unable to start queue mirror on node ''rabbit@rabbit-02''. Target virtual host is not running: {vhost_supervisor_not_running,<<"/">>}
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:43.736 [info] <0.629.0> Mirrored queue 'retry-05-32000ms' in vhost '/': Adding mirror on node 'rabbit@rabbit-03': <17407.715.0>
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:43.741 [info] <0.623.0> Mirrored queue 'retry-07-128000ms' in vhost '/': Adding mirror on node 'rabbit@rabbit-03': <17407.720.0>
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:43.808 [warning] <0.626.0> Mirrored queue 'retry-10-1024000ms' in vhost '/': Unable to start queue mirror on node ''rabbit@rabbit-02''. Target virtual host is not running: {vhost_supervisor_not_running,<<"/">>}
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:43.816 [warning] <0.638.0> Mirrored queue 'retry-06-64000ms' in vhost '/': Unable to start queue mirror on node ''rabbit@rabbit-02''. Target virtual host is not running: {vhost_supervisor_not_running,<<"/">>}
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:43.850 [info] <0.632.0> Mirrored queue 'retry-02-4000ms' in vhost '/': Adding mirror on node 'rabbit@rabbit-03': <17407.735.0>
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:43.859 [info] <0.635.0> Mirrored queue 'retry-03-8000ms' in vhost '/': Adding mirror on node 'rabbit@rabbit-03': <17407.740.0>
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:43.931 [warning] <0.644.0> Mirrored queue 'retry-11-2048000ms' in vhost '/': Unable to start queue mirror on node ''rabbit@rabbit-02''. Target virtual host is not running: {vhost_supervisor_not_running,<<"/">>}
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:43.932 [warning] <0.641.0> Mirrored queue 'retry-04-16000ms' in vhost '/': Unable to start queue mirror on node ''rabbit@rabbit-02''. Target virtual host is not running: {vhost_supervisor_not_running,<<"/">>}
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:43.969 [info] <0.626.0> Mirrored queue 'retry-10-1024000ms' in vhost '/': Adding mirror on node 'rabbit@rabbit-03': <17407.756.0>
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:43.975 [info] <0.638.0> Mirrored queue 'retry-06-64000ms' in vhost '/': Adding mirror on node 'rabbit@rabbit-03': <17407.759.0>
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:44.039 [warning] <0.650.0> Mirrored queue 'retry-01-2000ms' in vhost '/': Unable to start queue mirror on node ''rabbit@rabbit-02''. Target virtual host is not running: {vhost_supervisor_not_running,<<"/">>}
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:44.046 [warning] <0.647.0> Mirrored queue 'retry-09-512000ms' in vhost '/': Unable to start queue mirror on node ''rabbit@rabbit-02''. Target virtual host is not running: {vhost_supervisor_not_running,<<"/">>}
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:44.093 [info] <0.644.0> Mirrored queue 'retry-11-2048000ms' in vhost '/': Adding mirror on node 'rabbit@rabbit-03': <17407.776.0>
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:44.093 [info] <0.641.0> Mirrored queue 'retry-04-16000ms' in vhost '/': Adding mirror on node 'rabbit@rabbit-03': <17407.779.0>
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:44.156 [warning] <0.653.0> Mirrored queue 'retry-08-256000ms' in vhost '/': Unable to start queue mirror on node ''rabbit@rabbit-02''. Target virtual host is not running: {vhost_supervisor_not_running,<<"/">>}
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:44.163 [warning] <0.656.0> Mirrored queue 'retry-12-4096000ms' in vhost '/': Unable to start queue mirror on node ''rabbit@rabbit-02''. Target virtual host is not running: {vhost_supervisor_not_running,<<"/">>}
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:44.200 [info] <0.650.0> Mirrored queue 'retry-01-2000ms' in vhost '/': Adding mirror on node 'rabbit@rabbit-03': <17407.797.0>
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:44.213 [info] <0.647.0> Mirrored queue 'retry-09-512000ms' in vhost '/': Adding mirror on node 'rabbit@rabbit-03': <17407.800.0>
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:44.406 [info] <0.653.0> Mirrored queue 'retry-08-256000ms' in vhost '/': Adding mirror on node 'rabbit@rabbit-03': <17407.816.0>
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:44.407 [info] <0.656.0> Mirrored queue 'retry-12-4096000ms' in vhost '/': Adding mirror on node 'rabbit@rabbit-03': <17407.819.0>
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:46.019 [info] <0.791.0> queue 'processBroadcast' in vhost '/': granting vote for {'%2F_processBroadcast','rabbit@rabbit-02'} with last indexterm {418,1} for term 2 previous term was 1
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:46.041 [info] <0.791.0> queue 'processBroadcast' in vhost '/': detected a new leader {'%2F_processBroadcast','rabbit@rabbit-02'} in term 2
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:46.078 [info] <0.783.0> queue 'processEvent' in vhost '/': granting vote for {'%2F_processEvent','rabbit@rabbit-02'} with last indexterm {19420,1} for term 2 previous term was 1
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:46.119 [info] <0.783.0> queue 'processEvent' in vhost '/': detected a new leader {'%2F_processEvent','rabbit@rabbit-02'} in term 2
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:46.401 [info] <0.799.0> queue 'generateWebsite' in vhost '/': granting vote for {'%2F_generateWebsite','rabbit@rabbit-02'} with last indexterm {366,1} for term 2 previous term was 1
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:46.425 [info] <0.799.0> queue 'generateWebsite' in vhost '/': detected a new leader {'%2F_generateWebsite','rabbit@rabbit-02'} in term 2
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:46.479 [info] <0.819.0> queue 'retry' in vhost '/': granting vote for {'%2F_retry','rabbit@rabbit-02'} with last indexterm {1364,1} for term 2 previous term was 1
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:46.500 [info] <0.819.0> queue 'retry' in vhost '/': detected a new leader {'%2F_retry','rabbit@rabbit-02'} in term 2
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:46.640 [info] <0.808.0> queue 'sendEmail' in vhost '/': granting vote for {'%2F_sendEmail','rabbit@rabbit-02'} with last indexterm {366,1} for term 2 previous term was 2
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:46.665 [info] <0.808.0> queue 'sendEmail' in vhost '/': detected a new leader {'%2F_sendEmail','rabbit@rabbit-02'} in term 2
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:46.687 [info] <0.836.0> queue 'processBulkOperation' in vhost '/': granting vote for {'%2F_processBulkOperation','rabbit@rabbit-02'} with last indexterm {366,1} for term 2 previous term was 1
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:46.709 [info] <0.836.0> queue 'processBulkOperation' in vhost '/': detected a new leader {'%2F_processBulkOperation','rabbit@rabbit-02'} in term 2
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:46.956 [info] <0.827.0> queue 'processWorkflow' in vhost '/': granting vote for {'%2F_processWorkflow','rabbit@rabbit-02'} with last indexterm {1234,1} for term 2 previous term was 1
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:46.965 [info] <0.540.0> rabbit on node 'rabbit@rabbit-02' up
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:46.984 [info] <0.827.0> queue 'processWorkflow' in vhost '/': detected a new leader {'%2F_processWorkflow','rabbit@rabbit-02'} in term 2
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:47.011 [info] <0.847.0> queue 'sendTribesEmail' in vhost '/': granting vote for {'%2F_sendTribesEmail','rabbit@rabbit-02'} with last indexterm {482,1} for term 2 previous term was 1
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:47.034 [info] <0.847.0> queue 'sendTribesEmail' in vhost '/': detected a new leader {'%2F_sendTribesEmail','rabbit@rabbit-02'} in term 2
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:47.728 [info] <0.650.0> Mirrored queue 'retry-01-2000ms' in vhost '/': Synchronising: 0 messages to synchronise
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:47.728 [info] <0.647.0> Mirrored queue 'retry-09-512000ms' in vhost '/': Synchronising: 0 messages to synchronise
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:47.728 [info] <0.647.0> Mirrored queue 'retry-09-512000ms' in vhost '/': Synchronising: batch size: 4096
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:47.728 [info] <0.644.0> Mirrored queue 'retry-11-2048000ms' in vhost '/': Synchronising: 0 messages to synchronise
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:47.728 [info] <0.641.0> Mirrored queue 'retry-04-16000ms' in vhost '/': Synchronising: 0 messages to synchronise
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:47.729 [info] <0.635.0> Mirrored queue 'retry-03-8000ms' in vhost '/': Synchronising: 0 messages to synchronise
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:47.729 [info] <0.638.0> Mirrored queue 'retry-06-64000ms' in vhost '/': Synchronising: 0 messages to synchronise
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:47.729 [info] <0.632.0> Mirrored queue 'retry-02-4000ms' in vhost '/': Synchronising: 0 messages to synchronise
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:47.729 [info] <0.638.0> Mirrored queue 'retry-06-64000ms' in vhost '/': Synchronising: batch size: 4096
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:47.729 [info] <0.623.0> Mirrored queue 'retry-07-128000ms' in vhost '/': Synchronising: 0 messages to synchronise
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:47.729 [info] <0.650.0> Mirrored queue 'retry-01-2000ms' in vhost '/': Synchronising: batch size: 4096
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:47.730 [info] <0.629.0> Mirrored queue 'retry-05-32000ms' in vhost '/': Synchronising: 0 messages to synchronise
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:47.730 [info] <0.644.0> Mirrored queue 'retry-11-2048000ms' in vhost '/': Synchronising: batch size: 4096
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:47.730 [info] <0.629.0> Mirrored queue 'retry-05-32000ms' in vhost '/': Synchronising: batch size: 4096
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:47.730 [info] <0.641.0> Mirrored queue 'retry-04-16000ms' in vhost '/': Synchronising: batch size: 4096
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:47.730 [info] <0.635.0> Mirrored queue 'retry-03-8000ms' in vhost '/': Synchronising: batch size: 4096
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:47.730 [info] <0.632.0> Mirrored queue 'retry-02-4000ms' in vhost '/': Synchronising: batch size: 4096
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:47.731 [info] <0.623.0> Mirrored queue 'retry-07-128000ms' in vhost '/': Synchronising: batch size: 4096
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:47.844 [info] <0.877.0> Mirrored queue 'retry-11-2048000ms' in vhost '/': Synchronising: all mirrors already synced
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.024 [info] <0.872.0> Mirrored queue 'retry-09-512000ms' in vhost '/': Synchronising: all mirrors already synced
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.074 [info] <0.876.0> Mirrored queue 'retry-01-2000ms' in vhost '/': Synchronising: all mirrors already synced
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.123 [info] <0.875.0> Mirrored queue 'retry-03-8000ms' in vhost '/': Synchronising: all mirrors already synced
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.181 [info] <0.873.0> Mirrored queue 'retry-06-64000ms' in vhost '/': Synchronising: all mirrors already synced
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.524 [info] <0.303.0> Running boot step empty_db_check defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.524 [info] <0.303.0> Will not seed default virtual host and user: have definitions to load...
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.524 [info] <0.303.0> Running boot step rabbit_looking_glass defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.524 [info] <0.303.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.525 [info] <0.303.0> Running boot step background_gc defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.526 [info] <0.303.0> Running boot step routing_ready defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.526 [info] <0.303.0> Running boot step pre_flight defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.526 [info] <0.303.0> Running boot step notify_cluster defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.526 [info] <0.303.0> Running boot step networking defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.527 [info] <0.303.0> Running boot step definition_import_worker_pool defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.527 [info] <0.425.0> Starting worker pool 'definition_import_pool' with 2 processes in it
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.528 [info] <0.303.0> Running boot step cluster_name defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.529 [info] <0.303.0> Running boot step direct_client defined by app rabbit
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.532 [info] <0.303.0> Running boot step rabbit_management_load_definitions defined by app rabbitmq_management
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.532 [info] <0.901.0> Resetting node maintenance status
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.532 [info] <0.44.0> Application rabbit started on node 'rabbit@rabbit-01'
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.615 [info] <0.626.0> Mirrored queue 'retry-10-1024000ms' in vhost '/': Synchronising: 0 messages to synchronise
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.615 [info] <0.626.0> Mirrored queue 'retry-10-1024000ms' in vhost '/': Synchronising: batch size: 4096
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.739 [info] <0.635.0> Mirrored queue 'retry-03-8000ms' in vhost '/': Synchronising: 0 messages to synchronise
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.739 [info] <0.635.0> Mirrored queue 'retry-03-8000ms' in vhost '/': Synchronising: batch size: 4096
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.749 [info] <0.644.0> Mirrored queue 'retry-11-2048000ms' in vhost '/': Synchronising: 0 messages to synchronise
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.749 [info] <0.644.0> Mirrored queue 'retry-11-2048000ms' in vhost '/': Synchronising: batch size: 4096
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.751 [info] <0.874.0> Mirrored queue 'retry-05-32000ms' in vhost '/': Synchronising: all mirrors already synced
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.919 [info] <0.878.0> Mirrored queue 'retry-04-16000ms' in vhost '/': Synchronising: all mirrors already synced
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:48.963 [info] <0.540.0> rabbit on node 'rabbit@rabbit-03' up
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:49.334 [info] <0.641.0> Mirrored queue 'retry-04-16000ms' in vhost '/': Synchronising: 0 messages to synchronise
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:49.334 [info] <0.641.0> Mirrored queue 'retry-04-16000ms' in vhost '/': Synchronising: batch size: 4096
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:49.369 [info] <0.638.0> Mirrored queue 'retry-06-64000ms' in vhost '/': Synchronising: 0 messages to synchronise
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:49.369 [info] <0.638.0> Mirrored queue 'retry-06-64000ms' in vhost '/': Synchronising: batch size: 4096
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:49.871 [info] <0.540.0> rabbit on node 'rabbit@rabbit-02' up
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:49.881 [info] <0.44.0> Application rabbitmq_management_agent started on node 'rabbit@rabbit-01'
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:49.882 [info] <0.44.0> Application cowlib started on node 'rabbit@rabbit-01'
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:49.883 [info] <0.44.0> Application cowboy started on node 'rabbit@rabbit-01'
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:49.885 [info] <0.44.0> Application rabbitmq_web_dispatch started on node 'rabbit@rabbit-01'
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:49.885 [info] <0.44.0> Application amqp_client started on node 'rabbit@rabbit-01'
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:49.938 [info] <0.998.0> Management plugin: HTTP (non-TLS) listener started on port 15672
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:49.989 [info] <0.998.0> Management plugin: HTTPS listener started on port 15671
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:49.989 [info] <0.1209.0> Statistics database started.
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:49.990 [info] <0.1208.0> Starting worker pool 'management_worker_pool' with 3 processes in it
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:49.999 [info] <0.44.0> Application rabbitmq_management started on node 'rabbit@rabbit-01'
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:50.027 [info] <0.44.0> Application prometheus started on node 'rabbit@rabbit-01'
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:50.030 [info] <0.1226.0> Prometheus metrics: HTTP (non-TLS) listener started on port 15692
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:50.030 [info] <0.901.0> Ready to start client connection listeners
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:50.032 [info] <0.44.0> Application rabbitmq_prometheus started on node 'rabbit@rabbit-01'
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:50.036 [info] <0.1347.0> started TLS (SSL) listener on [::]:5671
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:50.347 [info] <0.540.0> rabbit on node 'rabbit@rabbit-03' up
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:50.623 [info] <0.901.0> Server startup complete; 4 plugins started.
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |  * rabbitmq_prometheus
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |  * rabbitmq_management
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |  * rabbitmq_web_dispatch
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |  * rabbitmq_management_agent
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    |  completed with 4 plugins.
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:50.623 [info] <0.901.0> Resetting node maintenance status
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:50.626 [info] <0.880.0> Mirrored queue 'retry-07-128000ms' in vhost '/': Synchronising: all mirrors already synced
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:50.626 [info] <0.623.0> Mirrored queue 'retry-07-128000ms' in vhost '/': Synchronising: 0 messages to synchronise
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:50.626 [info] <0.623.0> Mirrored queue 'retry-07-128000ms' in vhost '/': Synchronising: batch size: 4096
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:50.653 [info] <0.650.0> Mirrored queue 'retry-01-2000ms' in vhost '/': Synchronising: 0 messages to synchronise
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:50.653 [info] <0.650.0> Mirrored queue 'retry-01-2000ms' in vhost '/': Synchronising: batch size: 4096
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:51.179 [info] <0.911.0> Mirrored queue 'retry-10-1024000ms' in vhost '/': Synchronising: all mirrors already synced
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:51.179 [info] <0.626.0> Mirrored queue 'retry-10-1024000ms' in vhost '/': Synchronising: 0 messages to synchronise
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:51.180 [info] <0.656.0> Mirrored queue 'retry-12-4096000ms' in vhost '/': Synchronising: 0 messages to synchronise
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:51.180 [info] <0.626.0> Mirrored queue 'retry-10-1024000ms' in vhost '/': Synchronising: batch size: 4096
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:51.180 [info] <0.656.0> Mirrored queue 'retry-12-4096000ms' in vhost '/': Synchronising: batch size: 4096
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:51.901 [info] <0.1353.0> Mirrored queue 'retry-07-128000ms' in vhost '/': Synchronising: all mirrors already synced
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:51.927 [info] <0.916.0> Mirrored queue 'retry-03-8000ms' in vhost '/': Synchronising: all mirrors already synced
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:52.145 [info] <0.917.0> Mirrored queue 'retry-11-2048000ms' in vhost '/': Synchronising: all mirrors already synced
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:52.146 [info] <0.879.0> Mirrored queue 'retry-02-4000ms' in vhost '/': Synchronising: all mirrors already synced
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:52.147 [info] <0.653.0> Mirrored queue 'retry-08-256000ms' in vhost '/': Synchronising: 0 messages to synchronise
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:52.147 [info] <0.653.0> Mirrored queue 'retry-08-256000ms' in vhost '/': Synchronising: batch size: 4096
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:52.149 [info] <0.1354.0> Mirrored queue 'retry-01-2000ms' in vhost '/': Synchronising: all mirrors already synced
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:52.170 [info] <0.937.0> Mirrored queue 'retry-06-64000ms' in vhost '/': Synchronising: all mirrors already synced
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:52.173 [info] <0.936.0> Mirrored queue 'retry-04-16000ms' in vhost '/': Synchronising: all mirrors already synced
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:52.174 [info] <0.1362.0> Mirrored queue 'retry-10-1024000ms' in vhost '/': Synchronising: all mirrors already synced
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:52.178 [info] <0.1388.0> Mirrored queue 'retry-08-256000ms' in vhost '/': Synchronising: all mirrors already synced
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:52.180 [info] <0.653.0> Mirrored queue 'retry-08-256000ms' in vhost '/': Synchronising: 0 messages to synchronise
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:52.180 [info] <0.1363.0> Mirrored queue 'retry-12-4096000ms' in vhost '/': Synchronising: all mirrors already synced
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:52.180 [info] <0.629.0> Mirrored queue 'retry-05-32000ms' in vhost '/': Synchronising: 0 messages to synchronise
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:52.181 [info] <0.653.0> Mirrored queue 'retry-08-256000ms' in vhost '/': Synchronising: batch size: 4096
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:52.181 [info] <0.629.0> Mirrored queue 'retry-05-32000ms' in vhost '/': Synchronising: batch size: 4096
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:52.183 [info] <0.656.0> Mirrored queue 'retry-12-4096000ms' in vhost '/': Synchronising: 0 messages to synchronise
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:52.183 [info] <0.656.0> Mirrored queue 'retry-12-4096000ms' in vhost '/': Synchronising: batch size: 4096
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:52.213 [info] <0.1394.0> Mirrored queue 'retry-05-32000ms' in vhost '/': Synchronising: all mirrors already synced
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:52.213 [info] <0.1393.0> Mirrored queue 'retry-08-256000ms' in vhost '/': Synchronising: all mirrors already synced
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:52.214 [info] <0.647.0> Mirrored queue 'retry-09-512000ms' in vhost '/': Synchronising: 0 messages to synchronise
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:52.214 [info] <0.647.0> Mirrored queue 'retry-09-512000ms' in vhost '/': Synchronising: batch size: 4096
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:52.216 [info] <0.1395.0> Mirrored queue 'retry-12-4096000ms' in vhost '/': Synchronising: all mirrors already synced
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:52.217 [info] <0.632.0> Mirrored queue 'retry-02-4000ms' in vhost '/': Synchronising: 0 messages to synchronise
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:52.217 [info] <0.632.0> Mirrored queue 'retry-02-4000ms' in vhost '/': Synchronising: batch size: 4096
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:52.244 [info] <0.1396.0> Mirrored queue 'retry-09-512000ms' in vhost '/': Synchronising: all mirrors already synced
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:33:52.247 [info] <0.1397.0> Mirrored queue 'retry-02-4000ms' in vhost '/': Synchronising: all mirrors already synced
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:35:49.379 [info] <0.1796.0> accepting AMQP connection <0.1796.0> (10.0.4.249:35758 -> 10.0.4.244:5671)
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:35:49.434 [info] <0.1796.0> connection <0.1796.0> (10.0.4.249:35758 -> 10.0.4.244:5671): user 'ISBE5CLPgay5dA9eVrxsY0hADacDvlPjYQXvXd17wJvp6mEFTRthNIKhnkZddj' authenticated and granted access to vhost '/'
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:35:49.455 [info] <0.1805.0> quorum_queue declaring {amqqueue,{resource,<<47>>,queue,<<116,101,115,116,45,113,117,101,117,101>>},true,false,none,[{<<120,45,113,117,101,117,101,45,116,121,112,101>>,longstr,<<113,117,111,114,117,109>>}],none,[],[],[],undefined,undefined,[],[],live,0,[],<<47>>,#{user => <<73,83,66,69,53,67,76,80,103,97,121,53,100,65,57,101,86,114,120,115,89,48,104,65,68,97,99,68,118,108,80,106,89,81,88,118,88,100,49,55,119,74,118,112,54,109,69,70,84,82,116,104,78,73,75,104,110,107,90,100,100,106>>},rabbit_quorum_queue,#{}}
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:35:49.760 [notice] <0.1812.0> queue 'test-queue' in vhost '/': candidate -> leader in term: 1 machine version: 1
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 2021-03-23 11:35:49.764 [info] <0.1805.0> ra: started cluster %2F_test-queue with 3 servers
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | 0 servers failed to start: []
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | Leader: {'%2F_test-queue','rabbit@rabbit-01'}
rabbit_rabbit-01.1.ldlcbpe7wi1u@rabbit_FSN1_1    | {removed_failing_handler,rabbit_log}

[jaschaio]

I didn't change anything else in the original configuration (e.G. using environment variables + docker config) but the logging is working now even after the inital startup and I don't see the {removed_failing_handler,rabbit_log} error any longer.