Quorum queues for delayed queues to ensure *at-least-once* delivery

[philipheimboeck]

• This change requires at least RabbitMQ 3.10.
• Version used: 14.1

Hello everyone!

In a clustered environment it can happen that messages are being lost when using dead-letter-exchanges. Now this also means that when using delayed messages, it can happen that messages would get lost.
The documentation states this:

By default, dead-lettered messages are re-published without publisher confirms turned on internally. Therefore using DLX in a clustered RabbitMQ environment is not guaranteed to be safe.

https://www.rabbitmq.com/docs/dlx#safety

With quorum queues it is possible to enable at-least-once dead-lettering for a source quorum queue.

https://www.rabbitmq.com/docs/quorum-queues#dead-lettering

I suggest a change, where when the quorum option is enabled, the dead letter queues are also created as quorum queues and additionally enable the settings to for at-least-once delivery. The motivation behind quorum queues is to focus on data safety, so I think the same should apply to delayed queues.

Please note that this topic was also already discussed in #564, but it was closed due to not bringing any benefits. However I think the reasoning was a different one as it was about performance and not data safety.

I have tested the following patch locally with our system. Please note that I also disabled the x-expires option, which means the queues won't disappear anymore.

This is due to two reasons:

1. Quorum queues are not really for short-lived queues, therefore it makes more sense to keep them alive. Also in many use cases the delays will be the same many times, so there shouldn't be too many new queues. Of course this is a topic which has to be discussed and heavily depends on the use cases, but it might be still be an ok limitation if quorum queues are needed.
2. In my tests I pushed many messages to the queues and it happened that the client crashed, because a message was pushed while it was removed (tested with RabbitMQ 3.13.7). By removing this that problem was gone.

--- /dev/null
+++ ../src/Queue/RabbitMQQueue.php
@@ -625,11 +625,21 @@
      */
     protected function getDelayQueueArguments(string $destination, int $ttl): array
     {
+        $arguments = [];
+        $isQuorum = $this->getConfig()->isQuorum();
+        if ($isQuorum) {
+            $arguments = [
+                'x-queue-type' => 'quorum',
+                'x-dead-letter-strategy' => 'at-least-once',
+                'x-overflow' => 'reject-publish',
+            ];
+        }
         return [
+           ...$arguments,
             'x-dead-letter-exchange' => $this->getExchange(),
             'x-dead-letter-routing-key' => $this->getRoutingKey($destination),
             'x-message-ttl' => $ttl,
-            'x-expires' => $ttl * 2,
+           ...($isQuorum ? [] : ['x-expires' => $ttl * 2]),
         ];
     }