[stable/rabbitmq] Recover from "Waiting for Mnesia tables" after all nodes forced shutdown

[denis111]

Is your feature request related to a problem? Please describe.
Yes, in our dev/staging environment in AWS we turn-off at night time (set scaling group to 0) the nodes of EKS cluster to save costs and we enabled persistence for rabbitmq, so it was like unexpected shutdown for all rabbitmq nodes and it can't recover with "Waiting for Mnesia tables", i tried setting podManagementPolicy and service.alpha.kubernetes.io/tolerate-unready-endpoints: "true" but it had no effect, still keeps looping with "Waiting for Mnesia tables"

Describe the solution you'd like
Can this solution be applied #9645 (comment) as an option? We'd really prefer availability over integrity.

Describe alternatives you've considered
Don't use persistence.

[miguelaeh]

Hi @denis111 ,
Are you able to know the order in which the nodes were shut down? I am not a RabbitMQ expert but in the documentation you can see this:

Normally when you shut down a RabbitMQ cluster altogether, the first node you restart should be the last one to go down, since it may have seen things happen that other nodes did not. But sometimes that's not possible: for instance if the entire cluster loses power then all nodes may think they were not the last to shut down.

Link to documentation: https://www.rabbitmq.com/rabbitmqctl.8.html#force_boot

Could you try if the force_boot works in case you not know the order?

[denis111]

@miguelaeh Thank you for answering. No, we can't know the order, this is just autoscaling group with schedule to scale to 0 instances at night time when nobody's working. So it's unacceptable for us if anything is not able to recover from such "disaster" as sort of "unexpected" shut down of all nodes so in that case we prefer not using persistence because we prefer availability over integrity.

I hope I'll try to play with force_boot this friday and I will let you know if it worked.

[miguelaeh]

Thank you @denis111 ,
let me know what happen when you try with this option.

[denis111]

Well, first, I can't execute rabbitmqctl force_boot because it says "Error: this command requires the 'rabbit' app to be stopped on the target node. Stop it with 'rabbitmqctl stop_app'.". But if we stop it the pod will just restart without giving a possibility to execute "abbitmqctl force_boot"...
So i created force_load file in "/opt/bitnami/rabbitmq/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-pre-0.rabbitmq-pre-headless.pre.svc.cluster.local" in my case, and it worked!

[miguelaeh]

I'm glad it worked.
That is a cool solution.

[denis111]

Yes but how to autmate it? I mean the creation of force_load file in some init container maybe...

[miguelaeh]

You could try mounting the file in the init container via a Configmap, but if you don't need to execute any command before the main container start, I guess you could just mount the file in the main container (also with a Configmap).

[denis111]

I can't find the "helm way" to do it, existing configmap template in rabbitmq chart doesn't allow to add extra files as well as statefuset template doesn't allow to add some extra init container or extra volume mount...

[miguelaeh]

The Chart does not support that at this moment. You have to add it manually (you can clone the repository and modify the Chart with your needs),

[denis111]

Well, we'd like to use mainstream chart, I'll see if I can make pull request then.

[denis111]

I've detected if we enable forceBoot option on a new install without existing PVC (with clean new volume) then RabbitMQ is unable to start with Error: enoent. I'm creating a PR to address this issue.