Look into setting podManagementPolicy to Parallel

[ChunyiLyu]

Context

PodManagementPolicy defaults to OrderedReady . When set to Parallel statefulSet controller starts and deletes all pods in a statefulSet in parallel and does not wait for any pod to become Ready or Terminated.

Setting podManagementPolicy: Parallel does not impact updates, since maxUnavailable is default to 1.

This is only a suggestion. It's likely that starting up all rabbitmq nodes in parallel just works. However we are unclear about other possible side effects.

Additional context
Add any other context or screenshots about the feature request here.

[ChunyiLyu]

fyi @mkuratczyk

It's not a urgent improvement, we can pick it up when we have time.

[mkuratczyk]

Discussion on this topic: https://rabbitmq.slack.com/archives/CTMSV81HA/p1600173056054400

[github-actions]

This issue has been marked as stale due to 60 days of inactivity. Stale issues will be closed after a further 30 days of inactivity; please remove the stale label in order to prevent this occurring.

[ChunyiLyu]

I did some brief testing with podManagementPolicy set to Parallel and with randomized startup delay set to

cluster_formation.randomized_startup_delay_range.min = 20
cluster_formation.randomized_startup_delay_range.max = 60

1. Creation and deletion are a lot faster!
2. Rolling upgrade works and stays the same behavior as OrderedReady . This is because the operator sets spec.updateStrategy to RollingUpdate with partition 0.
3. Recreating all pods at once. This is the scenario where users had reported problem with podManagementPolicy: OrderedReady in https://rabbitmq.slack.com/archives/CTMSV81HA/p1600173056054400. When using Parallel, pods would sometimes crash (see below for boot failed error log)and restart automatically. I suspect it happened when the order of pods coming back up is not the desired order. When it restarts again, it will succeed because most of the other pods have already joined the cluster successfully so the order no long matters. Though restart is needed in some cases, it's a better situation than using OrderedReady because restarts are done automatically by k8s and there's no human intervention required. However if there are unconsumed messages that's not persistent to disks during this time, restarts would lead to data loss.

Configuring logger redirection
09:08:01.818 [error]

09:08:01.822 [error] BOOT FAILED
BOOT FAILED
09:08:01.822 [error] ===========
09:08:01.823 [error] ERROR: epmd error for host sample-server-3.sample-nodes.rabbitmq-system: timeout (timed out)
===========
ERROR: epmd error for host sample-server-3.sample-nodes.rabbitmq-system: timeout (timed out)
09:08:01.823 [error]

09:08:02.823 [error] Supervisor rabbit_prelaunch_sup had child prelaunch started with rabbit_prelaunch:run_prelaunch_first_phase() at undefined exit with reason {epmd_error,"sample-server-3.sample-nodes.rabbitmq-system",timeout} in context start_error
09:08:02.824 [error] CRASH REPORT Process <0.156.0> with 0 neighbours exited with reason: {{shutdown,{failed_to_start_child,prelaunch,{epmd_error,"sample-server-3.sample-nodes.rabbitmq-system",timeout}}},{rabbit_prelaunch_app,start,[normal,[]]}} in application_master:init/4 line 138
{"Kernel pid terminated",application_controller,"{application_start_failure,rabbitmq_prelaunch,{{shutdown,{failed_to_start_child,prelaunch,{epmd_error,\"sample-server-3.sample-nodes.rabbitmq-system\",timeout}}},{rabbit_prelaunch_app,start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,rabbitmq_prelaunch,{{shutdown,{failed_to_start_child,prelaunch,{epmd_error,"sample-server-3.sample-nodes.rabbitmq-system",tim

Crash dump is being written to: erl_crash.dump...%

[michaelklishin]

rabbit_prelaunch:run_prelaunch_first_phase/0 exits with an error trying to contact EPMD. This means that either the port is not yet available, or the hostname does not yet resolve, or something of that kind.

[ChunyiLyu]

Context dump.

Other than the epmd timeout error. We have also seen two other errors

Configuring logger redirection
2021-01-13 14:55:34.162 [debug] <0.287.0> Lager installed handler error_logger_lager_h into error_logger
2021-01-13 14:55:34.170 [debug] <0.296.0> Lager installed handler lager_forwarder_backend into rabbit_log_channel_lager_event
2021-01-13 14:55:34.170 [debug] <0.320.0> Lager installed handler lager_forwarder_backend into rabbit_log_ra_lager_event
2021-01-13 14:55:34.171 [debug] <0.323.0> Lager installed handler lager_forwarder_backend into rabbit_log_shovel_lager_event
2021-01-13 14:55:34.171 [debug] <0.326.0> Lager installed handler lager_forwarder_backend into rabbit_log_upgrade_lager_event
2021-01-13 14:55:34.171 [debug] <0.290.0> Lager installed handler lager_forwarder_backend into error_logger_lager_event
2021-01-13 14:55:34.171 [debug] <0.293.0> Lager installed handler lager_forwarder_backend into rabbit_log_lager_event
2021-01-13 14:55:34.171 [debug] <0.302.0> Lager installed handler lager_forwarder_backend into rabbit_log_feature_flags_lager_event
2021-01-13 14:55:34.171 [debug] <0.305.0> Lager installed handler lager_forwarder_backend into rabbit_log_federation_lager_event
2021-01-13 14:55:34.171 [debug] <0.299.0> Lager installed handler lager_forwarder_backend into rabbit_log_connection_lager_event
2021-01-13 14:55:34.171 [debug] <0.308.0> Lager installed handler lager_forwarder_backend into rabbit_log_ldap_lager_event
2021-01-13 14:55:34.171 [debug] <0.311.0> Lager installed handler lager_forwarder_backend into rabbit_log_mirroring_lager_event
2021-01-13 14:55:34.171 [debug] <0.317.0> Lager installed handler lager_forwarder_backend into rabbit_log_queue_lager_event
2021-01-13 14:55:34.171 [debug] <0.314.0> Lager installed handler lager_forwarder_backend into rabbit_log_prelaunch_lager_event
2021-01-13 14:55:34.190 [info] <0.44.0> Application lager started on node 'rabbit@one-server-0.one-nodes.rabbitmq-system'
2021-01-13 14:55:34.662 [debug] <0.283.0> Lager installed handler lager_backend_throttle into lager_event
2021-01-13 14:55:36.309 [info] <0.44.0> Application mnesia started on node 'rabbit@one-server-0.one-nodes.rabbitmq-system'
2021-01-13 14:55:36.310 [info] <0.272.0> 
 Starting RabbitMQ 3.8.9 on Erlang 23.1.5
 Copyright (c) 2007-2020 VMware, Inc. or its affiliates.
 Licensed under the MPL 2.0. Website: https://rabbitmq.com

  ##  ##      RabbitMQ 3.8.9
  ##  ##
  ##########  Copyright (c) 2007-2020 VMware, Inc. or its affiliates.
  ######  ##
  ##########  Licensed under the MPL 2.0. Website: https://rabbitmq.com

  Doc guides: https://rabbitmq.com/documentation.html
  Support:    https://rabbitmq.com/contact.html
  Tutorials:  https://rabbitmq.com/getstarted.html
  Monitoring: https://rabbitmq.com/monitoring.html

  Logs: <stdout>

  Config file(s): /etc/rabbitmq/rabbitmq.conf
                  /etc/rabbitmq/conf.d/default_user.conf

  Starting broker...2021-01-13 14:55:36.311 [info] <0.272.0> 
 node           : rabbit@one-server-0.one-nodes.rabbitmq-system
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/rabbitmq/rabbitmq.conf
                : /etc/rabbitmq/conf.d/default_user.conf
 cookie hash    : 850r16/uIB+MQQFVHCy1gQ==
 log(s)         : <stdout>
 database dir   : /var/lib/rabbitmq/mnesia/rabbit@one-server-0.one-nodes.rabbitmq-system
2021-01-13 14:55:36.321 [info] <0.272.0> Running boot step pre_boot defined by app rabbit
2021-01-13 14:55:36.321 [info] <0.272.0> Running boot step rabbit_core_metrics defined by app rabbit
2021-01-13 14:55:36.322 [info] <0.272.0> Running boot step rabbit_alarm defined by app rabbit
2021-01-13 14:55:36.327 [info] <0.375.0> Memory high watermark set to 6417 MiB (6729654272 bytes) of 16044 MiB (16824135680 bytes) total
2021-01-13 14:55:36.331 [info] <0.377.0> Enabling free disk space monitoring
2021-01-13 14:55:36.332 [info] <0.377.0> Disk free limit set to 2000MB
2021-01-13 14:55:36.338 [info] <0.272.0> Running boot step code_server_cache defined by app rabbit
2021-01-13 14:55:36.338 [info] <0.272.0> Running boot step file_handle_cache defined by app rabbit
2021-01-13 14:55:36.339 [info] <0.395.0> FHC read buffering:  OFF
2021-01-13 14:55:36.339 [info] <0.395.0> FHC write buffering: ON
2021-01-13 14:55:36.339 [info] <0.272.0> Running boot step worker_pool defined by app rabbit
2021-01-13 14:55:36.340 [info] <0.370.0> Will use 4 processes for default worker pool
2021-01-13 14:55:36.340 [info] <0.370.0> Starting worker pool 'worker_pool' with 4 processes in it
2021-01-13 14:55:36.340 [info] <0.394.0> Limiting to approx 1048479 file handles (943629 sockets)
2021-01-13 14:55:36.340 [info] <0.272.0> Running boot step database defined by app rabbit
2021-01-13 14:55:36.343 [info] <0.272.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-01-13 14:55:36.360 [info] <0.272.0> Successfully synced tables from a peer
2021-01-13 14:55:36.360 [warning] <0.272.0> Feature flags: the previous instance of this node must have failed to write the `feature_flags` file at `/var/lib/rabbitmq/mnesia/rabbit@one-server-0.one-nodes.rabbitmq-system-feature_flags`:
2021-01-13 14:55:36.360 [warning] <0.272.0> Feature flags:   - list of previously disabled feature flags now marked as such: [drop_unroutable_metric,empty_basic_get_metric]
2021-01-13 14:55:36.368 [info] <0.272.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-01-13 14:55:36.394 [info] <0.272.0> Successfully synced tables from a peer
2021-01-13 14:55:36.418 [info] <0.272.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-01-13 14:55:36.418 [info] <0.272.0> Successfully synced tables from a peer
2021-01-13 14:55:36.432 [info] <0.272.0> Will register with peer discovery backend rabbit_peer_discovery_k8s
2021-01-13 14:55:36.507 [info] <0.272.0> Running boot step database_sync defined by app rabbit
2021-01-13 14:55:36.507 [info] <0.272.0> Running boot step feature_flags defined by app rabbit
2021-01-13 14:55:36.507 [info] <0.272.0> Running boot step codec_correctness_check defined by app rabbit
2021-01-13 14:55:36.507 [info] <0.272.0> Running boot step external_infrastructure defined by app rabbit
2021-01-13 14:55:36.507 [info] <0.272.0> Running boot step rabbit_registry defined by app rabbit
2021-01-13 14:55:36.508 [info] <0.272.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2021-01-13 14:55:36.508 [info] <0.272.0> Running boot step rabbit_queue_location_random defined by app rabbit
2021-01-13 14:55:36.508 [info] <0.272.0> Running boot step rabbit_event defined by app rabbit
2021-01-13 14:55:36.508 [info] <0.272.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2021-01-13 14:55:36.508 [info] <0.272.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2021-01-13 14:55:36.508 [info] <0.272.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2021-01-13 14:55:36.508 [info] <0.272.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2021-01-13 14:55:36.508 [info] <0.272.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2021-01-13 14:55:36.509 [info] <0.272.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2021-01-13 14:55:36.509 [info] <0.272.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2021-01-13 14:55:36.509 [info] <0.272.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2021-01-13 14:55:36.509 [info] <0.272.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2021-01-13 14:55:36.509 [info] <0.272.0> Running boot step rabbit_priority_queue defined by app rabbit
2021-01-13 14:55:36.509 [info] <0.272.0> Priority queues enabled, real BQ is rabbit_variable_queue
2021-01-13 14:55:36.509 [info] <0.272.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2021-01-13 14:55:36.509 [info] <0.272.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2021-01-13 14:55:36.510 [info] <0.272.0> Running boot step kernel_ready defined by app rabbit
2021-01-13 14:55:36.510 [info] <0.272.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2021-01-13 14:55:36.510 [info] <0.272.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2021-01-13 14:55:36.512 [info] <0.445.0> epmd monitor knows us, inter-node communication (distribution) port: 25672
2021-01-13 14:55:36.512 [info] <0.272.0> Running boot step guid_generator defined by app rabbit
2021-01-13 14:55:36.518 [info] <0.272.0> Running boot step rabbit_node_monitor defined by app rabbit
2021-01-13 14:55:36.518 [info] <0.449.0> Starting rabbit_node_monitor
2021-01-13 14:55:36.518 [info] <0.272.0> Running boot step delegate_sup defined by app rabbit
2021-01-13 14:55:36.519 [info] <0.272.0> Running boot step rabbit_memory_monitor defined by app rabbit
2021-01-13 14:55:36.519 [info] <0.272.0> Running boot step core_initialized defined by app rabbit
2021-01-13 14:55:36.519 [info] <0.272.0> Running boot step upgrade_queues defined by app rabbit
2021-01-13 14:55:36.546 [info] <0.272.0> Running boot step rabbit_connection_tracking defined by app rabbit
2021-01-13 14:55:36.546 [info] <0.272.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2021-01-13 14:55:36.546 [info] <0.272.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2021-01-13 14:55:36.546 [info] <0.272.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2021-01-13 14:55:36.547 [info] <0.272.0> Running boot step rabbit_policies defined by app rabbit
2021-01-13 14:55:36.548 [info] <0.272.0> Running boot step rabbit_policy defined by app rabbit
2021-01-13 14:55:36.548 [info] <0.272.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2021-01-13 14:55:36.548 [info] <0.272.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit
2021-01-13 14:55:36.548 [info] <0.272.0> Running boot step rabbit_vhost_limit defined by app rabbit
2021-01-13 14:55:36.548 [info] <0.272.0> Running boot step recovery defined by app rabbit
2021-01-13 14:55:36.550 [info] <0.478.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@one-server-0.one-nodes.rabbitmq-system/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2021-01-13 14:55:36.556 [info] <0.478.0> Starting message stores for vhost '/'
2021-01-13 14:55:36.556 [info] <0.482.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2021-01-13 14:55:36.567 [info] <0.478.0> Started message store of type transient for vhost '/'
2021-01-13 14:55:36.568 [info] <0.486.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2021-01-13 14:55:36.573 [info] <0.478.0> Started message store of type persistent for vhost '/'
2021-01-13 14:55:36.575 [info] <0.272.0> Running boot step empty_db_check defined by app rabbit
2021-01-13 14:55:36.575 [info] <0.272.0> Will not seed default virtual host and user: have definitions to load...
2021-01-13 14:55:36.575 [info] <0.272.0> Running boot step rabbit_looking_glass defined by app rabbit
2021-01-13 14:55:36.575 [info] <0.272.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2021-01-13 14:55:36.575 [info] <0.272.0> Running boot step background_gc defined by app rabbit
2021-01-13 14:55:36.575 [info] <0.272.0> Running boot step connection_tracking defined by app rabbit
2021-01-13 14:55:36.576 [info] <0.272.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_rabbit@one-server-0.one-nodes.rabbitmq-system'
2021-01-13 14:55:36.576 [info] <0.272.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_rabbit@one-server-0.one-nodes.rabbitmq-system'
2021-01-13 14:55:36.576 [info] <0.272.0> Running boot step routing_ready defined by app rabbit
2021-01-13 14:55:36.576 [info] <0.272.0> Running boot step pre_flight defined by app rabbit
2021-01-13 14:55:36.576 [info] <0.272.0> Running boot step notify_cluster defined by app rabbit
2021-01-13 14:55:36.576 [info] <0.272.0> Running boot step networking defined by app rabbit
2021-01-13 14:55:36.576 [info] <0.272.0> Running boot step definition_import_worker_pool defined by app rabbit
2021-01-13 14:55:36.577 [info] <0.370.0> Starting worker pool 'definition_import_pool' with 4 processes in it
2021-01-13 14:55:36.577 [info] <0.272.0> Running boot step cluster_name defined by app rabbit
2021-01-13 14:55:36.577 [info] <0.272.0> Setting cluster name to 'one' as configured
2021-01-13 14:55:36.581 [info] <0.272.0> Running boot step direct_client defined by app rabbit
2021-01-13 14:55:36.581 [info] <0.44.0> Application rabbit started on node 'rabbit@one-server-0.one-nodes.rabbitmq-system'
2021-01-13 14:55:37.153 [info] <0.517.0> Feature flags: list of feature flags found:
2021-01-13 14:55:37.153 [info] <0.517.0> Feature flags:   [x] drop_unroutable_metric
2021-01-13 14:55:37.153 [info] <0.517.0> Feature flags:   [x] empty_basic_get_metric
2021-01-13 14:55:37.153 [info] <0.517.0> Feature flags:   [x] implicit_default_bindings
2021-01-13 14:55:37.153 [info] <0.517.0> Feature flags:   [x] maintenance_mode_status
2021-01-13 14:55:37.153 [info] <0.517.0> Feature flags:   [x] quorum_queue
2021-01-13 14:55:37.153 [info] <0.517.0> Feature flags:   [x] virtual_host_metadata
2021-01-13 14:55:37.153 [info] <0.517.0> Feature flags: feature flag states written to disk: yes
2021-01-13 14:55:37.392 [info] <0.517.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2021-01-13 14:55:37.392 [info] <0.517.0> Management plugin: using rates mode 'basic'
2021-01-13 14:55:37.401 [info] <0.44.0> Application rabbitmq_management_agent started on node 'rabbit@one-server-0.one-nodes.rabbitmq-system'
2021-01-13 14:55:37.424 [info] <0.44.0> Application cowlib started on node 'rabbit@one-server-0.one-nodes.rabbitmq-system'
2021-01-13 14:55:37.448 [info] <0.44.0> Application cowboy started on node 'rabbit@one-server-0.one-nodes.rabbitmq-system'
2021-01-13 14:55:37.468 [info] <0.44.0> Application rabbitmq_web_dispatch started on node 'rabbit@one-server-0.one-nodes.rabbitmq-system'
2021-01-13 14:55:37.485 [info] <0.44.0> Application amqp_client started on node 'rabbit@one-server-0.one-nodes.rabbitmq-system'
2021-01-13 14:55:37.502 [info] <0.517.0> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2021-01-13 14:55:37.502 [info] <0.517.0> Running boot step rabbit_management_load_definitions defined by app rabbitmq_management
2021-01-13 14:55:37.542 [info] <0.582.0> Management plugin: HTTP (non-TLS) listener started on port 15672
2021-01-13 14:55:37.542 [info] <0.688.0> Statistics database started.
2021-01-13 14:55:37.542 [info] <0.687.0> Starting worker pool 'management_worker_pool' with 3 processes in it
2021-01-13 14:55:37.544 [info] <0.44.0> Application rabbitmq_management started on node 'rabbit@one-server-0.one-nodes.rabbitmq-system'
2021-01-13 14:55:37.594 [info] <0.44.0> Application prometheus started on node 'rabbit@one-server-0.one-nodes.rabbitmq-system'
2021-01-13 14:55:37.612 [info] <0.703.0> Peer discovery: node cleanup is disabled
2021-01-13 14:55:37.612 [info] <0.44.0> Application rabbitmq_peer_discovery_common started on node 'rabbit@one-server-0.one-nodes.rabbitmq-system'
2021-01-13 14:55:37.629 [info] <0.44.0> Application rabbitmq_peer_discovery_k8s started on node 'rabbit@one-server-0.one-nodes.rabbitmq-system'
2021-01-13 14:55:37.650 [info] <0.711.0> Prometheus metrics: HTTP (non-TLS) listener started on port 15692
2021-01-13 14:55:37.650 [info] <0.517.0> Ready to start client connection listeners
2021-01-13 14:55:37.650 [info] <0.44.0> Application rabbitmq_prometheus started on node 'rabbit@one-server-0.one-nodes.rabbitmq-system'
2021-01-13 14:56:09.287 [error] <0.517.0> 
2021-01-13 14:56:09.287 [error] <0.517.0> BOOT FAILED

BOOT FAILED
===========
Error during startup: {error,no_epmd_port}
2021-01-13 14:56:09.287 [error] <0.517.0> ===========
2021-01-13 14:56:09.288 [error] <0.517.0> Error during startup: {error,no_epmd_port}
2021-01-13 14:56:09.288 [error] <0.517.0> 

2021-01-13 14:56:10.377 [info] <0.517.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbitmq_management
    amqp_client
    rabbitmq_prometheus
    rabbitmq_web_dispatch
    cowboy
    cowlib
    rabbitmq_management_agent
    rabbitmq_peer_discovery_k8s
    rabbitmq_peer_discovery_common
    rabbit
    rabbitmq_prelaunch
    rabbit_common
    prometheus
    sysmon_handler
    os_mon
    ra
    mnesia
2021-01-13 14:56:10.377 [info] <0.517.0> Stopping application 'rabbitmq_management'
2021-01-13 14:56:10.380 [warning] <0.574.0> HTTP listener registry could not find context rabbitmq_management_tls
2021-01-13 14:56:10.382 [info] <0.517.0> Stopping application 'amqp_client'
2021-01-13 14:56:10.382 [info] <0.44.0> Application rabbitmq_management exited with reason: stopped
2021-01-13 14:56:10.382 [info] <0.44.0> Application rabbitmq_management exited with reason: stopped
2021-01-13 14:56:10.385 [info] <0.517.0> Stopping application 'rabbitmq_prometheus'
2021-01-13 14:56:10.385 [info] <0.44.0> Application amqp_client exited with reason: stopped
2021-01-13 14:56:10.385 [info] <0.44.0> Application amqp_client exited with reason: stopped
2021-01-13 14:56:10.388 [warning] <0.574.0> HTTP listener registry could not find context rabbitmq_prometheus_tls
2021-01-13 14:56:10.390 [info] <0.517.0> Stopping application 'rabbitmq_web_dispatch'
2021-01-13 14:56:10.390 [info] <0.44.0> Application rabbitmq_prometheus exited with reason: stopped
2021-01-13 14:56:10.391 [info] <0.44.0> Application rabbitmq_prometheus exited with reason: stopped
2021-01-13 14:56:10.393 [info] <0.517.0> Stopping application 'cowboy'
2021-01-13 14:56:10.393 [info] <0.44.0> Application rabbitmq_web_dispatch exited with reason: stopped
2021-01-13 14:56:10.394 [info] <0.44.0> Application rabbitmq_web_dispatch exited with reason: stopped
2021-01-13 14:56:10.396 [info] <0.44.0> Application cowboy exited with reason: stopped
2021-01-13 14:56:10.396 [info] <0.517.0> Stopping application 'cowlib'
2021-01-13 14:56:10.396 [info] <0.44.0> Application cowboy exited with reason: stopped
2021-01-13 14:56:10.396 [info] <0.517.0> Stopping application 'rabbitmq_management_agent'
2021-01-13 14:56:10.397 [info] <0.44.0> Application cowlib exited with reason: stopped
2021-01-13 14:56:10.397 [info] <0.44.0> Application cowlib exited with reason: stopped
2021-01-13 14:56:10.400 [info] <0.517.0> Stopping application 'rabbitmq_peer_discovery_k8s'
2021-01-13 14:56:10.400 [info] <0.44.0> Application rabbitmq_management_agent exited with reason: stopped
2021-01-13 14:56:10.400 [info] <0.44.0> Application rabbitmq_management_agent exited with reason: stopped
2021-01-13 14:56:10.403 [info] <0.517.0> Stopping application 'rabbitmq_peer_discovery_common'
2021-01-13 14:56:10.403 [info] <0.44.0> Application rabbitmq_peer_discovery_k8s exited with reason: stopped
2021-01-13 14:56:10.403 [info] <0.44.0> Application rabbitmq_peer_discovery_k8s exited with reason: stopped
2021-01-13 14:56:10.406 [info] <0.44.0> Application rabbitmq_peer_discovery_common exited with reason: stopped
2021-01-13 14:56:10.406 [info] <0.517.0> Stopping application 'rabbit'
2021-01-13 14:56:10.406 [info] <0.44.0> Application rabbitmq_peer_discovery_common exited with reason: stopped
2021-01-13 14:56:10.406 [info] <0.272.0> Will unregister with peer discovery backend rabbit_peer_discovery_k8s
2021-01-13 14:56:10.406 [info] <0.470.0> Closing all connections in vhost '/' on node 'rabbit@one-server-0.one-nodes.rabbitmq-system' because the vhost is stopping
2021-01-13 14:56:10.406 [info] <0.486.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/rabbit@one-server-0.one-nodes.rabbitmq-system/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent'
2021-01-13 14:56:10.414 [info] <0.486.0> Message store for directory '/var/lib/rabbitmq/mnesia/rabbit@one-server-0.one-nodes.rabbitmq-system/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' is stopped
2021-01-13 14:56:10.414 [info] <0.482.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/rabbit@one-server-0.one-nodes.rabbitmq-system/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient'
2021-01-13 14:56:10.419 [info] <0.482.0> Message store for directory '/var/lib/rabbitmq/mnesia/rabbit@one-server-0.one-nodes.rabbitmq-system/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped
2021-01-13 14:56:10.428 [info] <0.44.0> Application rabbit exited with reason: stopped
2021-01-13 14:56:10.428 [info] <0.44.0> Application rabbit exited with reason: stopped
{"init terminating in do_boot",{error,no_epmd_port}}
init terminating in do_boot ({error,no_epmd_port})

Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done

Configuring logger redirection
15:15:49.778 [error] 

BOOT FAILED
===========
ERROR: epmd error for host one-server-0.one-nodes.rabbitmq-system: address (cannot connect to host/port)

15:15:49.783 [error] BOOT FAILED
15:15:49.784 [error] ===========
15:15:49.784 [error] ERROR: epmd error for host one-server-0.one-nodes.rabbitmq-system: address (cannot connect to host/port)
15:15:49.784 [error] 
15:15:50.785 [error] Supervisor rabbit_prelaunch_sup had child prelaunch started with rabbit_prelaunch:run_prelaunch_first_phase() at undefined exit with reason {epmd_error,"one-server-0.one-nodes.rabbitmq-system",address} in context start_error
15:15:50.785 [error] CRASH REPORT Process <0.156.0> with 0 neighbours exited with reason: {{shutdown,{failed_to_start_child,prelaunch,{epmd_error,"one-server-0.one-nodes.rabbitmq-system",address}}},{rabbit_prelaunch_app,start,[normal,[]]}} in application_master:init/4 line 138
{"Kernel pid terminated",application_controller,"{application_start_failure,rabbitmq_prelaunch,{{shutdown,{failed_to_start_child,prelaunch,{epmd_error,\"one-server-0.one-nodes.rabbitmq-system\",address}}},{rabbit_prelaunch_app,start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,rabbitmq_prelaunch,{{shutdown,{failed_to_start_child,prelaunch,{epmd_error,"one-server-0.one-nodes.rabbitmq-system",address}}

Crash dump is being written to: erl_crash.dump...

Though the cause for these different errors are likely the same.

[Gsantomaggio]

The problem happens when the Erlang Port Mapping tires to get the nodes here

During the issue, the epmd actually works epmd -names gives the right result, but:

$ erl -sname test
net_adm:names('tls-server-1.tls-nodes.default'). <-- get blocked

by using the full dns name it works:

 net_adm:names('tls-server-1.tls-nodes.default.svc.cluster.local').

I recall that we used the full DNS name when we wrote this example because of a similar problem.

I changed the operator using:

{
Name:  "RABBITMQ_NODENAME",
Value: "rabbit@$(MY_POD_NAME).$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE).svc.cluster.local",
},
{
Name:  "K8S_HOSTNAME_SUFFIX",
Value: ".$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE).svc.cluster.local",
},

and seems to works because the EPMD can resolve always the running nodes.
Of course, this is not a solution just a workaround.

Last note: the pod dies during the rabbitmq_prelaunch node after this timeout and not during the rabbitmq node

[Gsantomaggio]

I created a temp docker image: gsantomaggio/rabbitmq:3.8.13.epdm with the Retry epmd name lookups N times during pre-launch fix.

Use the following definition to test it:

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name:  test
spec:
  replicas: 3
  override:
    statefulSet:
      spec:
        template:
          spec:
            containers:
              - image: gsantomaggio/rabbitmq:3.8.13.epdm
                name: rabbitmq
                imagePullPolicy: Always

After the first tests seem to work correctly.

NOTE: gsantomaggio/rabbitmq:3.8.13.epdm is only for testing and will be deleted soon.

[mkuratczyk]

Great stuff Gabriele! I've been killing pods all morning and didn't get a single crash. :)