Speed up fanout exchange #14546

ansd · 2025-09-15T15:30:00Z

Resolves #14531
Resolves #14533

What?

Increase end-to-end message throughput for messages routed via the fanout exchange by ~42% (see benchmark below).
In addition to the fanout exchange, a similar speed up is achieved for the following exchange types:

modulus hash
random
recent history

This applies only if Khepri is enabled.

How?

Use an additional routing table (projection) rabbit_khepri_route_by_source whose table key is the source exchange.
Looking up the destinations happens then by an ETS table key.

Prior to this commit, CPUs were busy compiling the same match spec for every incoming message.

For the bug fix of #14533, we also introduce a new projection called rabbit_khepri_route_by_source_key. This can be thought of a v2 of the old rabbit_khepri_index_route projection. The old rabbit_khepri_index_route projection gets deleted in the rabbitmq_4.2.0 feature flag callback. The two new projections are registered at boot time (as discussed in #14543 (comment))

Benchmark

Start RabbitMQ:

make run-broker RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+S 5" \
    RABBITMQ_CONFIG_FILE="advanced.config" PLUGINS="rabbitmq_management"

where advanced.config contains:

[
 {rabbitmq_management_agent, [
  {disable_metrics_collector, true}
 ]}
].

Create a queue and binding:

deps/rabbitmq_management/bin/rabbitmqadmin declare queue queue_type=classic durable=true name=q1 && \
deps/rabbitmq_management/bin/rabbitmqadmin declare binding source=amq.fanout destination=q1

Create the load

java -jar target/perf-test.jar -p -e amq.fanout -u q1 -s 5 --autoack -z 60

Before this commit:

sending rate avg: 97394 msg/s
receiving rate avg: 97394 msg/s

After this commit:

sending rate avg: 138677 msg/s
receiving rate avg: 138677 msg/s

The CPU flamegraph shows that rabbit_exchange:route/3 consumes the following CPU amounts:

13.5% before this commit
3.4% after this commit

Downsides

Additional ETS memory usage for the new projection table.
However, the new table does not store any binding entries for the following
source exchange types:

direct
headers
topic
x-local-random
x-jms-topic

Also, care must be taken to not add extensive amounts of bindings with the same source exchange to projection rabbit_khepri_route_by_source. For fanout exchanges, binding an extensive amount of queues (e.g. thousands) to the same fanout exchanges is nonsensical anyway. One edge case could be binding thousands of queues to the random exchange type. But even such an exotic use case could be realised efficiently by using intermediate random exchanges (i.e. exchange-to-exchange bindings) 🙂

This commit adds a test case for a regression/bug that occurs in Khepri. ``` make -C deps/rabbit ct-bindings t=cluster:binding_args RABBITMQ_METADATA_STORE=mnesia ``` succeeds, but ``` make -C deps/rabbit ct-bindings t=cluster:binding_args RABBITMQ_METADATA_STORE=khepri ``` fails. The problem is that ETS table `rabbit_khepri_index_route` cannot differentiate between two bindings with different binding arguments, and therefore deletes entries too early, leading to wrong routing decisions. The solution to this bug is to include the binding arguments in the `rabbit_khepri_index_route` projection, similar to how the binding args are also included in the `rabbit_index_route` Mnesia table. This bug/regression is an edge case and exists if the source exchange type is `direct` or `fanout` and if different bindings arguments are used by client apps. Note that such binding arguments are entirely ignored when RabbitMQ performs routing decisions for the `direct` or `fanout` exchange. However, there might be client apps that use binding arguments to add some metadata to the binding, for example `app-id` or `user` or `purpose` and might use this metadata as a form of reference counting in deciding when to delete `auto-delete` exchanges or just for informational/operational purposes.

Fix #14533

Resolves #14531 ## What? Increase end-to-end message throughput for messages routed via the fanout exchange by ~42% (see benchmark below). In addition to the fanout exchange, a similar speed up is achieved for the following exchange types: * modulus hash * random * recent history This applies only if Khepri is enabled. ## How? Use an additional routing table (projection) whose table key is the source exchange. Looking up the destinations happens then by an ETS table key. Prior to this commit, CPUs were busy compiling the same match spec for every incoming message. ## Benchmark 1. Start RabbitMQ: ``` make run-broker RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+S 5" \ RABBITMQ_CONFIG_FILE="advanced.config" PLUGINS="rabbitmq_management" ``` where `advanced.config` contains: ``` [ {rabbitmq_management_agent, [ {disable_metrics_collector, true} ]} ]. ``` 2. Create a queue and binding: ``` deps/rabbitmq_management/bin/rabbitmqadmin declare queue queue_type=classic durable=true name=q1 && \ deps/rabbitmq_management/bin/rabbitmqadmin declare binding source=amq.fanout destination=q1 ``` 3. Create the load ``` java -jar target/perf-test.jar -p -e amq.fanout -u q1 -s 5 --autoack -z 60 ``` Before this commit: ``` sending rate avg: 97394 msg/s receiving rate avg: 97394 msg/s ``` After this commit: ``` sending rate avg: 138677 msg/s receiving rate avg: 138677 msg/s ``` The CPU flamegraph shows that `rabbit_exchange:route/3` consumes the following CPU amounts: * 13.5% before this commit * 3.4% after this commit ## Downsides Additional ETS memory usage for the new projection table. However, the new table does not store any binding entries for the following source exchange types: * direct * headers * topic * x-local-random

Test that exchange bindings work correctly with the new projection tables `rabbit_khepri_route_by_source` and `rabbit_khepri_route_by_source_key`.

deps/rabbit/src/rabbit_khepri.erl

Khepri won’t modify a projection that is already registered (based on its name).

deps/rabbit/src/rabbit_db_binding.erl

See #11667 (comment) for rationale.

the-mikedavis

👍

I see similar results locally for that perf-test scenario. main:

id: test-100516-125, sending rate avg: 95139 msg/s
id: test-100516-125, receiving rate avg: 95139 msg/s

This PR:

id: test-100853-180, sending rate avg: 140581 msg/s
id: test-100853-180, receiving rate avg: 140581 msg/s

ansd · 2025-09-17T15:17:06Z

As always, thanks a lot to both of you @dumbbell @the-mikedavis for your thorough reviews!

* Add test case for binding args Khepri regression This commit adds a test case for a regression/bug that occurs in Khepri. ``` make -C deps/rabbit ct-bindings t=cluster:binding_args RABBITMQ_METADATA_STORE=mnesia ``` succeeds, but ``` make -C deps/rabbit ct-bindings t=cluster:binding_args RABBITMQ_METADATA_STORE=khepri ``` fails. The problem is that ETS table `rabbit_khepri_index_route` cannot differentiate between two bindings with different binding arguments, and therefore deletes entries too early, leading to wrong routing decisions. The solution to this bug is to include the binding arguments in the `rabbit_khepri_index_route` projection, similar to how the binding args are also included in the `rabbit_index_route` Mnesia table. This bug/regression is an edge case and exists if the source exchange type is `direct` or `fanout` and if different bindings arguments are used by client apps. Note that such binding arguments are entirely ignored when RabbitMQ performs routing decisions for the `direct` or `fanout` exchange. However, there might be client apps that use binding arguments to add some metadata to the binding, for example `app-id` or `user` or `purpose` and might use this metadata as a form of reference counting in deciding when to delete `auto-delete` exchanges or just for informational/operational purposes. * Fix regression with Khepri binding args Fix #14533 * Speed up fanout exchange Resolves #14531 ## What? Increase end-to-end message throughput for messages routed via the fanout exchange by ~42% (see benchmark below). In addition to the fanout exchange, a similar speed up is achieved for the following exchange types: * modulus hash * random * recent history This applies only if Khepri is enabled. ## How? Use an additional routing table (projection) whose table key is the source exchange. Looking up the destinations happens then by an ETS table key. Prior to this commit, CPUs were busy compiling the same match spec for every incoming message. ## Benchmark 1. Start RabbitMQ: ``` make run-broker RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+S 5" \ RABBITMQ_CONFIG_FILE="advanced.config" PLUGINS="rabbitmq_management" ``` where `advanced.config` contains: ``` [ {rabbitmq_management_agent, [ {disable_metrics_collector, true} ]} ]. ``` 2. Create a queue and binding: ``` deps/rabbitmq_management/bin/rabbitmqadmin declare queue queue_type=classic durable=true name=q1 && \ deps/rabbitmq_management/bin/rabbitmqadmin declare binding source=amq.fanout destination=q1 ``` 3. Create the load ``` java -jar target/perf-test.jar -p -e amq.fanout -u q1 -s 5 --autoack -z 60 ``` Before this commit: ``` sending rate avg: 97394 msg/s receiving rate avg: 97394 msg/s ``` After this commit: ``` sending rate avg: 138677 msg/s receiving rate avg: 138677 msg/s ``` The CPU flamegraph shows that `rabbit_exchange:route/3` consumes the following CPU amounts: * 13.5% before this commit * 3.4% after this commit ## Downsides Additional ETS memory usage for the new projection table. However, the new table does not store any binding entries for the following source exchange types: * direct * headers * topic * x-local-random * Add exchange binding tests Test that exchange bindings work correctly with the new projection tables `rabbit_khepri_route_by_source` and `rabbit_khepri_route_by_source_key`. * Always register all projections Khepri won’t modify a projection that is already registered (based on its name). * Protect ets:lookup_element/4 in try catch See #11667 (comment) for rationale. (cherry picked from commit a66c716)

* Add test case for binding args Khepri regression This commit adds a test case for a regression/bug that occurs in Khepri. ``` make -C deps/rabbit ct-bindings t=cluster:binding_args RABBITMQ_METADATA_STORE=mnesia ``` succeeds, but ``` make -C deps/rabbit ct-bindings t=cluster:binding_args RABBITMQ_METADATA_STORE=khepri ``` fails. The problem is that ETS table `rabbit_khepri_index_route` cannot differentiate between two bindings with different binding arguments, and therefore deletes entries too early, leading to wrong routing decisions. The solution to this bug is to include the binding arguments in the `rabbit_khepri_index_route` projection, similar to how the binding args are also included in the `rabbit_index_route` Mnesia table. This bug/regression is an edge case and exists if the source exchange type is `direct` or `fanout` and if different bindings arguments are used by client apps. Note that such binding arguments are entirely ignored when RabbitMQ performs routing decisions for the `direct` or `fanout` exchange. However, there might be client apps that use binding arguments to add some metadata to the binding, for example `app-id` or `user` or `purpose` and might use this metadata as a form of reference counting in deciding when to delete `auto-delete` exchanges or just for informational/operational purposes. * Fix regression with Khepri binding args Fix #14533 * Speed up fanout exchange Resolves #14531 ## What? Increase end-to-end message throughput for messages routed via the fanout exchange by ~42% (see benchmark below). In addition to the fanout exchange, a similar speed up is achieved for the following exchange types: * modulus hash * random * recent history This applies only if Khepri is enabled. ## How? Use an additional routing table (projection) whose table key is the source exchange. Looking up the destinations happens then by an ETS table key. Prior to this commit, CPUs were busy compiling the same match spec for every incoming message. ## Benchmark 1. Start RabbitMQ: ``` make run-broker RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+S 5" \ RABBITMQ_CONFIG_FILE="advanced.config" PLUGINS="rabbitmq_management" ``` where `advanced.config` contains: ``` [ {rabbitmq_management_agent, [ {disable_metrics_collector, true} ]} ]. ``` 2. Create a queue and binding: ``` deps/rabbitmq_management/bin/rabbitmqadmin declare queue queue_type=classic durable=true name=q1 && \ deps/rabbitmq_management/bin/rabbitmqadmin declare binding source=amq.fanout destination=q1 ``` 3. Create the load ``` java -jar target/perf-test.jar -p -e amq.fanout -u q1 -s 5 --autoack -z 60 ``` Before this commit: ``` sending rate avg: 97394 msg/s receiving rate avg: 97394 msg/s ``` After this commit: ``` sending rate avg: 138677 msg/s receiving rate avg: 138677 msg/s ``` The CPU flamegraph shows that `rabbit_exchange:route/3` consumes the following CPU amounts: * 13.5% before this commit * 3.4% after this commit ## Downsides Additional ETS memory usage for the new projection table. However, the new table does not store any binding entries for the following source exchange types: * direct * headers * topic * x-local-random * Add exchange binding tests Test that exchange bindings work correctly with the new projection tables `rabbit_khepri_route_by_source` and `rabbit_khepri_route_by_source_key`. * Always register all projections Khepri won’t modify a projection that is already registered (based on its name). * Protect ets:lookup_element/4 in try catch See #11667 (comment) for rationale. (cherry picked from commit a66c716) Co-authored-by: David Ansari <david.ansari@gmx.de>

(cherry picked from commit b3a58b8)

* Add test case for binding args Khepri regression This commit adds a test case for a regression/bug that occurs in Khepri. ``` make -C deps/rabbit ct-bindings t=cluster:binding_args RABBITMQ_METADATA_STORE=mnesia ``` succeeds, but ``` make -C deps/rabbit ct-bindings t=cluster:binding_args RABBITMQ_METADATA_STORE=khepri ``` fails. The problem is that ETS table `rabbit_khepri_index_route` cannot differentiate between two bindings with different binding arguments, and therefore deletes entries too early, leading to wrong routing decisions. The solution to this bug is to include the binding arguments in the `rabbit_khepri_index_route` projection, similar to how the binding args are also included in the `rabbit_index_route` Mnesia table. This bug/regression is an edge case and exists if the source exchange type is `direct` or `fanout` and if different bindings arguments are used by client apps. Note that such binding arguments are entirely ignored when RabbitMQ performs routing decisions for the `direct` or `fanout` exchange. However, there might be client apps that use binding arguments to add some metadata to the binding, for example `app-id` or `user` or `purpose` and might use this metadata as a form of reference counting in deciding when to delete `auto-delete` exchanges or just for informational/operational purposes. * Fix regression with Khepri binding args Fix #14533 * Speed up fanout exchange Resolves #14531 ## What? Increase end-to-end message throughput for messages routed via the fanout exchange by ~42% (see benchmark below). In addition to the fanout exchange, a similar speed up is achieved for the following exchange types: * modulus hash * random * recent history This applies only if Khepri is enabled. ## How? Use an additional routing table (projection) whose table key is the source exchange. Looking up the destinations happens then by an ETS table key. Prior to this commit, CPUs were busy compiling the same match spec for every incoming message. ## Benchmark 1. Start RabbitMQ: ``` make run-broker RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+S 5" \ RABBITMQ_CONFIG_FILE="advanced.config" PLUGINS="rabbitmq_management" ``` where `advanced.config` contains: ``` [ {rabbitmq_management_agent, [ {disable_metrics_collector, true} ]} ]. ``` 2. Create a queue and binding: ``` deps/rabbitmq_management/bin/rabbitmqadmin declare queue queue_type=classic durable=true name=q1 && \ deps/rabbitmq_management/bin/rabbitmqadmin declare binding source=amq.fanout destination=q1 ``` 3. Create the load ``` java -jar target/perf-test.jar -p -e amq.fanout -u q1 -s 5 --autoack -z 60 ``` Before this commit: ``` sending rate avg: 97394 msg/s receiving rate avg: 97394 msg/s ``` After this commit: ``` sending rate avg: 138677 msg/s receiving rate avg: 138677 msg/s ``` The CPU flamegraph shows that `rabbit_exchange:route/3` consumes the following CPU amounts: * 13.5% before this commit * 3.4% after this commit ## Downsides Additional ETS memory usage for the new projection table. However, the new table does not store any binding entries for the following source exchange types: * direct * headers * topic * x-local-random * Add exchange binding tests Test that exchange bindings work correctly with the new projection tables `rabbit_khepri_route_by_source` and `rabbit_khepri_route_by_source_key`. * Always register all projections Khepri won’t modify a projection that is already registered (based on its name). * Protect ets:lookup_element/4 in try catch See #11667 (comment) for rationale.

ansd self-assigned this Sep 15, 2025

ansd added the backport-v4.2.x label Sep 15, 2025

ansd force-pushed the speed-up-fanout-exchange branch from 9f8112c to 36b0954 Compare September 15, 2025 15:30

michaelklishin added this to the 4.3.0 milestone Sep 15, 2025

ansd force-pushed the speed-up-fanout-exchange branch 3 times, most recently from a1a6bbc to 1c1c91f Compare September 16, 2025 10:47

ansd mentioned this pull request Sep 16, 2025

Fix regression with Khepri binding args #14543

Closed

ansd added 4 commits September 16, 2025 15:37

Fix regression with Khepri binding args

17244de

Fix #14533

Add exchange binding tests

4876315

Test that exchange bindings work correctly with the new projection tables `rabbit_khepri_route_by_source` and `rabbit_khepri_route_by_source_key`.

ansd force-pushed the speed-up-fanout-exchange branch from 1c1c91f to 4876315 Compare September 16, 2025 13:37

ansd marked this pull request as ready for review September 16, 2025 13:39

ansd commented Sep 16, 2025

View reviewed changes

deps/rabbit/src/rabbit_khepri.erl Show resolved Hide resolved

ansd requested review from dumbbell and the-mikedavis September 16, 2025 13:48

dumbbell requested changes Sep 16, 2025

View reviewed changes

deps/rabbit/src/rabbit_khepri.erl Outdated Show resolved Hide resolved

deps/rabbit/src/rabbit_khepri.erl Show resolved Hide resolved

Always register all projections

63313e1

Khepri won’t modify a projection that is already registered (based on its name).

ansd requested a review from dumbbell September 16, 2025 16:09

the-mikedavis reviewed Sep 16, 2025

View reviewed changes

deps/rabbit/src/rabbit_db_binding.erl Outdated Show resolved Hide resolved

Protect ets:lookup_element/4 in try catch

af61e93

See #11667 (comment) for rationale.

ansd requested a review from the-mikedavis September 16, 2025 16:38

dumbbell approved these changes Sep 17, 2025

View reviewed changes

the-mikedavis approved these changes Sep 17, 2025

View reviewed changes

ansd merged commit a66c716 into main Sep 17, 2025
568 of 569 checks passed

ansd deleted the speed-up-fanout-exchange branch September 17, 2025 15:17

mergify bot mentioned this pull request Sep 17, 2025

Speed up fanout exchange (backport #14546) #14563

Merged

ansd added a commit that referenced this pull request Sep 18, 2025

Include #14546 to the 4.2.0 release notes

b3a58b8

ansd added a commit that referenced this pull request Sep 18, 2025

Include #14546 to the 4.2.0 release notes

5a16693

(cherry picked from commit b3a58b8)

michaelklishin pushed a commit that referenced this pull request Sep 24, 2025

Include #14546 to the 4.2.0 release notes

e005cd3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Speed up fanout exchange #14546

Speed up fanout exchange #14546

Uh oh!

ansd commented Sep 15, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

the-mikedavis left a comment

Uh oh!

ansd commented Sep 17, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Speed up fanout exchange #14546

Speed up fanout exchange #14546

Uh oh!

Conversation

ansd commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What?

How?

Benchmark

Downsides

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

the-mikedavis left a comment

Choose a reason for hiding this comment

Uh oh!

ansd commented Sep 17, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ansd commented Sep 15, 2025 •

edited

Loading