[autothrottle] throttling only when reassignment uses zookeeper?

[seanlutner]

Using kafka-kit to perform some rebuild operations to expand a broker cluster.

Process:

• I ran metricsfetcher to generate metrics
• I generated new maps for all my topics using topicmappr.
• I started autothrottle
• I take the generated maps and pass them to kafka-reassign-partitions --execute --reassignment-json-file my_topic.json --zookeeper zookeeper01:2181

Doing this works and when necessary autothrottle kicks in and does its thing. However, I was experimenting with using this command

kafka-reassign-partitions --execute --reassignment-json-file my_topic.json --bootstrap-server kafka01:9092 --additional in a loop to process batches of new maps. When I do this autothrottle seems to be unaware of what's going on and never kicks in. Is that expected? Am I doing something incorrectly?

[jamiealquiza]

The way autothrottle works is that it periodically checks for a reassignment existing in ZooKeeper. My guess is that the reassignment finishes before it sees it or if you have Kafka configured in a chroot in ZooKeeper, autothrottle must be configured to look there via the -zk-prefix flag.

If you have any autothrottle logs, that may help as well.

[jamiealquiza]

Also, I just read this again. What version of Kafka are you running? My guess here is that newer tools have ported the feature to the Kafka admin client, so yeah, it's possible that it's registering the reassignment in the controller. I'll have to read a bit more into this.