Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# BP-66: support throttling for zookeeper read during rereplication

### Motivation

Each time the cluster triggers the re-replication, all replicators will read data from zookeeper. This can cause a great
pressure on Zookeeper. We need to support throttling for zookeeper read of re-replication.

For example, in a Pulsar cluster, we enable auto-recovery for every bookie. There are 400 bookies in a cluster, which
means there are 400 replicators in the cluster.
And there are about 3000 ledgers in each bookie, 1/3 of them are small ledgers, whose size is less than 0.1MB, that is
1000 small ledgers in each bookie.
If one bookie is decommissioned, the read latency of zookeeper will increase to minutes.

### Configuration

add the following configuration:

```
zkReplicationTaskRateLimit
```

default value is 0, which means no limit.
Value greater than 0 will enable the rate limit, and the value is the number of tasks that can be acquired per second.
Decimals are allowed too, for example, 0.5 means 1 task every 2 seconds, 1 means 1 task per second, 2 means 2 tasks per
second, and so on.

### Proposed Changes

Add a new configuration `zkReplicationTaskRateLimit` to control the rate of zookeeper read of re-replication.

### Compatibility, Deprecation, and Migration Plan

Full compatibility with the existing behavior. No deprecation needed.