Add rate limit support for Replicator between clusters #4273
Conversation
|
run integration tests |
There was a problem hiding this comment.
Overall looks good to me, just some simple ideas that I want to discuss together.
Considering that we have many features need dispatch rate limit in the future and limiter can effect different levels
Features:
- Consumption
- Reader
- Replicator
Effect levels
- Tenant
- Namespace
- Topic
- Subscription(Consumption only)
Difference for all scenarios is the definition of DispatchRate. This will correspond to the configuration of different brokers and policies.
Leaving the common part in DispatchRateLimiter and allow users to specify the configuration and polices will be more scalable.
|
@codelipenghui thanks for the comments, It is a good idea, we may need consider more for DispatchRateLimiter in the future. Would you please open an issue to track your idea there? |
There was a problem hiding this comment.
The change LGTM. Just very minor comment.
@rdhabalia Please take a look as well.
...r-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentReplicator.java
Show resolved
Hide resolved
sijie
added
area/broker
type/feature
|
ping @rdhabalia |
### Motivation Some parameters are added in the `broker.conf` and `standalone.conf` files. However, those parameters are not updated in the docs. See the following PRs for details: #4150, #4066, #4197, #3819, #4261, #4273, #4320. ### Modifications Add those parameter info, and sync docs with the code. Does not update the description quite much, there are two reasons for this: 1. Keep doc content consistent with code. We need to update the description for those parameters in the code first, and then sync them in docs. 2. Will adopt a generator to generate those content automatically in the near future.
### Motivation Some parameters are added in the `broker.conf` and `standalone.conf` files. However, those parameters are not updated in the docs. See the following PRs for details: apache#4150, apache#4066, apache#4197, apache#3819, apache#4261, apache#4273, apache#4320. ### Modifications Add those parameter info, and sync docs with the code. Does not update the description quite much, there are two reasons for this: 1. Keep doc content consistent with code. We need to update the description for those parameters in the code first, and then sync them in docs. 2. Will adopt a generator to generate those content automatically in the near future.
Currently the rate limit between replication clusters is not able to control.
In Geo-replication, once a cluster is offline, and after a long time, if it comes back, it may get a lot of messages from other clusters, and may use all of the network bandwidth.
This PR tries to provided a way to control the rate limit between cluster replications. Add rate limit support for Replicator between clusters.
changes: