Some classic queues don't have consumers after node restart

[mkuratczyk]

@acogoluegnes this may or may not be a perf-test issue. To be investigated further next week.

Given a bunch of classic queues with publishers and consumers
when one of the nodes in the cluster is restarted
some queues no longer have consumers

This was originally reported in the context of a Kubernetes Operator and a full rolling restart. However, seems like it can be reproduced locally with just a single node restart.

Steps:

$ make start-cluster PLUGINS="rabbitmq_management"

$ perf-test -x 10 -y 10 -r 10 -c 10 -A 10 -q 10 -qq -qp 'qq-%d' -qpf 0 -qpt 9 -H amqp://guest:guest@localhost:5672,amqp://guest:guest@localhost:5673,amqp://guest:guest@localhost:5674

$ perf-test -x 10 -y 10 -r 10 -c 10 -A 10 -q 10 -qp 'cq-%d' -qpf 0 -qpt 9 --auto-delete false -f persistent -H amqp://guest:guest@localhost:5672,amqp://guest:guest@localhost:5673,amqp://guest:guest@localhost:5674

$ ./sbin/rabbitmqctl -n rabbit-1 stop_app; ./sbin/rabbitmqctl -n rabbit-1 start_app

Before restarting the node, the state of the queues looks like this:

After the restart, it looks like this:

Notes:

• some of the queues don't have consumers, it may or may not be a perf-test re-connection issue
• it seems to always work well for QQs; sometimes it takes a moment to "heal" - a queue or two will take longer to get a consumer but they do get it; meanwhile, some CQs just don't
• the issue doesn't occur in a single-node RabbitMQ (just replace start-cluster with run-broker and all is good)

In my tests, perf-test logs this error:

Caused by: com.rabbitmq.client.ShutdownSignalException: channel error;
  protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND
  - queue 'cq-9' in vhost '/' process is stopped by supervisor, class-id=60, method-id=20)

The original issue report also contains this:

Caused by: com.rabbitmq.client.ShutdownSignalException: channel error;
  protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND
  - failed to perform operation on queue 'perf-test-10q-rate100-5' in vhost 'perf-test-grid' due to timeout, class-id=60, method-id=20)

Originally reported on Slack: https://rabbitmq.slack.com/archives/CTMSV81HA/p1650538594525179
There are many additional details and potential issues. The above doesn't match exactly what was reported but is a simplified case of one of the issues that rules out mirroring/auto-delete/non-durable properties as the cause. The above uses durable, non-mirrored queues and still has the same behaviour of "losing consumers".

[acogoluegnes]

PerfTest tries to recover the consumers on the node that went down, but the queues do not exist anymore (basic.consume calls get 404), so it gives up immediately. Of course the queues show up again when the node restarts but that's too late. Note the queue without consumers are the ones on the first node, the one which restarted.

I would not say that PerfTest failure is by design, but it is a reasonable default behavior for a performance tool. What is it supposed to do when the queues do not exist anymore? It assumes they are gone for good, which is fair, but they are not. It could retry, but with what kind of back-off and timeout strategies? Should we introduce new options for this? That's a lot of questions for non-HA queues.

@mkuratczyk WDYT?

[lukebakken]

Could PerfTest support the java client's topology recovery feature?

[acogoluegnes]

PerfTest does support connection and topology recovery. But the steps are the following in this configuration: some consumers are consumers from queues on node 1, the consumers are connected to node 1 and the classic (non-mirrored) queues are on node 1 as well. Node 1 goes down, the consumers reconnect to node 2 or node 3, try to basic.consume to their respective queues, but they get a 404, because the queue do no longer exist and they cannot be re-created, so PerfTest stop trying for this consumers. Then node 1 comes back and its queues as well.

PerfTest stops the consumers because of the 404 error, in this case some retry logic would work, hence my remarks above.

[lukebakken]

I see, I missed the part where the queues aren't mirrored.

[mkuratczyk]

It sounds to me like perf-test's behavior is reasonable but I wonder whether we could do better. I see two options:

1. Perhaps returning 404 for a queue that exists but is not accessible is incorrect? Unfortunately it's been like that for a long time and I don't see any AMQP codes that would be more accurate. Having said that, the fact that for example rabbitmqadmin declare queue name=cq returns Not found because the queue already exists (on a node that is down), is confusing at best...
2. If we can't really change the above, then just like we account for this in error messages (channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - home node 'rabbit-1@mkuratczyk-a02' of durable queue 'cq' in vhost '/' is down or inaccessible - notice "or inaccessible"), perhaps perf-test should try to reconnect, because we know that 404 is a transient error (it may either go away when a node starts again or if the queue gets declared later).

[acogoluegnes]

2. If we can't really change the above, then just like we account for this in error messages (channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - home node 'rabbit-1@mkuratczyk-a02' of durable queue 'cq' in vhost '/' is down or inaccessible - notice "or inaccessible"), perhaps perf-test should try to reconnect, because we know that 404 is a transient error (it may either go away when a node starts again or if the queue gets declared later).

Good point, it should be possible to check the content of the message and make PerfTest retry. I'll have a look.