Default memory-high watermark might be too high

[phunehehe]

In a production workload we are observing that RabbitMQ can use up to 5 times the memory-high watermark.

Reading https://www.rabbitmq.com/memory.html and looking at the resource usage, it seems to me this could happen when RabbitMQ pages queue contents out to disk. In the graph below you can see that the number of messages (blue) increases gradually, while the amount of free memory (orange) dropped abruptly.

For this setup we are running a cluster with 2 nodes, each having 8GB of RAM. The memory-high watermark is configured at 10%. At the bottom of the valley, RabbitMQ was using ~ 4GB of RAM.

Assuming a somewhat linear relationship between the watermark and the peak memory usage, the watermark is about 20% of the peak. If RabbitMQ is to use 75% of RAM at its peak, the watermark should be at (75% * 20% =) 15%. Could somebody validate my thoughts here?

Another way to put this is, the GC or paging activities could be improved, but I don't know enough about RabbitMQ to say what is what.

[phunehehe]

As I wrote a new peak appeared at 6.3GB. At this point I'm going to just stick to 10%...

[michaelklishin]

Thank you for a suggestion that's based on actual observations. You can reduce the paging ratio instead if this is the case you're hitting. Or you could use lazy queues. And even then there are things that consume RAM (connections, channels) that paging cannot help with.

There are no plans to change the default. Different values accommodate different workloads and environments, and we have no way to measure what's most common for our users.