Classic mirrored queue leadership transfer eagerly clears the list of mirrors which can result in data loss

[ChunyiLyu]

Describe the bug

Data loss issue with mirror queues on operator deployed clusters. Reported by internal team. cc @mkuratczyk @Gsantomaggio

To Reproduce

Steps to reproduce the behavior:

1. Deploy 3 nodes rabbitmq cluster
2. Create HA policy with params: "ha-mode: exactly, ha-params: 2, ha-promote-on-failure: when-synced, ha-sync-batch-size: 5000, ha-sync-mode: automatic"
3. Use perf tests to populate the cluster with unconsumed messages -y0 -queue-pattern 'perf-test-%d' \ --auto-delete false --queue-pattern-from 1 --queue-pattern-to 10 -s 1000 -x 10 -y 0 -r 1000 -R 0 -f persistent
4. Stop perf-tests (without triggering any alarms and all queues should be synced)
5. Delete one node which has queue masters located on and observe data loss (0 message left)

Include any YAML or manifest necessary to reproduce the problem.

Expected behavior

After deleting one rabbitmq node, the mirror queue should be promoted as the master queue and rabbitmq should create a new mirror in node that's not deleted.

Screenshots

Version and environment information

• RabbitMQ: 3.8.9
• RabbitMQ Cluster Operator: Latest main: rabbitmq/cluster-operator@c420661
• Kubernetes: 1.17.0
• Cloud provider or hardware configuration:

Additional context

This is a reported issue. The user has also seem similar data loss with "ha-params: 3" on 5 nodes cluster.

[davidmort]

Hi All, i think this issue has been raised off the back of some testing we have been doing with our rabbitmq cluster, it seems to be happening once the rabbitmq-upgrade drain -t %d part of the preStop hook executes. i've overidden the preStop hook to run rabbitmqctl stop_app instead of the drain command and the problem seems to go away, im not sure of the side effects of using stop_app vs the drain command though 🤔.

please let me know if there is any more info i can provide, happy to help 👍

[mkuratczyk]

Thank you, yes, we can confirm we observed the same behaviour. Seems like a serious issue with the drain command - we were able to provision a cluster, load data and just run rabbitmq-upgrade drain on one node and observe data loss (we also tested this with other deployment methods, not just the Operator, and the results are the same). We are actively working on this problem and will keep you posted.

[mkuratczyk]

The issue is in the implementation of the maintenance mode (rabbitmq-upgrade drain command). For classic mirrored queues it selects a new leader randomly from the pool of all other cluster nodes and then deletes existing mirrors. Since the synchronization is async, if a node with no replica is picked, existing mirrors can get deleted before synchronization completes. We are discussing potential solutions, such as:

1. explicitly waiting for the synchronization to finish
2. selecting the new leader from the pool of nodes that already have replicas

The issue only affects classic mirrored queues with ha-mode set to exactly and ha-params set to a value lower than the number of nodes in the cluster.

Until the fix is available, some workarounds:

• not using the drain command if using queues that meet the criteria above
• using quorum queues (they are not affected because the leadership transfer is implemented as leader election and only existing queue cluster members participate in the election)
• replicating to all nodes in the cluster - this way whatever node is picked, it will have a replica