Cluster Formation not working as expected

[aaguilartablada]

Describe the bug

Sometimes the RabbitMQCluster objects create a cluster, but often they create standalone nodes.

To Reproduce

After installing 8 clusters as follows:

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: <name>
spec:
  replicas: 3
  resources:
    requests:
      cpu: 250m
      memory: 1Gi
    limits:
      cpu: 250m
      memory: 1Gi
  rabbitmq:
    additionalConfig: |
      cluster_partition_handling = pause_minority
      vm_memory_high_watermark_paging_ratio = 0.99
      disk_free_limit.relative = 1.0
      collect_statistics_interval = 10000
  persistence:
    storageClassName: default
    storage: "32Gi"

Only 3 of them are 3-nodes clusters. The 5 remaining are 2-nodes and 1-node clusters. This behavior seems to be completely random.

For example:

[scrm-az@localhost ~]$ kubectl -n test get pod -l app.kubernetes.io/name=one
NAME           READY   STATUS    RESTARTS   AGE
one-server-0   1/1     Running   0          35m
one-server-1   1/1     Running   0          35m
one-server-2   1/1     Running   0          35m

[scrm-az@localhost ~]$ kubectl -n test exec -it one-server-0 -- rabbitmqctl cluster_status
Cluster status of node rabbit@one-server-0.one-nodes.test ...
Basics

Cluster name: one

Disk Nodes

rabbit@one-server-0.one-nodes.test
rabbit@one-server-1.one-nodes.test

Running Nodes

rabbit@one-server-0.one-nodes.test
rabbit@one-server-1.one-nodes.test

[scrm-az@localhost ~]$ kubectl -n test exec -it one-server-2 -- rabbitmqctl cluster_status
Cluster status of node rabbit@one-server-2.one-nodes.test ...
Basics

Cluster name: one

Disk Nodes

rabbit@one-server-2.one-nodes.test

Running Nodes

rabbit@one-server-2.one-nodes.test

Expected behavior
I expect having eight 3-nodes clusters.

Version and environment information

• RabbitMQ: 3.8.27-debian-10-r49
• RabbitMQ Cluster Operator: 1.12.1-scratch-r0
• Kubernetes: v1.22.6
• Cloud provider: Azure AKS

UPDATE

Overriding StatefulSet spec to set podManagementPolicy as OrderedReady solve the problem. I relaunched the 8 clusters several times and we had 100% success forming clusters.

[Zerpet]

Thank you for the report! We did some work in rabbitmq/rabbitmq-server#3075 to fix this problem, it looks like the culprit may be the pod management policy, which I suspect changed after we merged that PR.

[aaguilartablada]

Asesome! Thanks for the info

[mkuratczyk]

Hi. We use Parallel for a reason - with OrderedReady, the cluster usually won't start if you lose all the pods at the same time (RabbitMQ requires the last node to disappear to start first, since it has the most recent state, but with OrderedReady that means you get a deadlock, unless you are lucky and server-0 was the last to shut down). Therefore, I'd not consider this the solution. Can I ask you to reproduce the problem and share logs from all nodes so that we can have a look? Also, would you mind trying with the rabbitmq image to see if it happens as well? I've just deployed 20 5-node clusters with 3.8 and 3.9 and every single time they correctly formed a cluster.

[github-actions]

This issue has been marked as stale due to 60 days of inactivity. Stale issues will be closed after a further 30 days of inactivity; please remove the stale label in order to prevent this occurring.

[github-actions]

Closing stale issue due to further inactivity.