Skip to content

Commit bd57882

Browse files
committed
[SPARK-20603][SS][TEST] Set default number of topic partitions to 1 to reduce the load
## What changes were proposed in this pull request? I checked the logs of https://amplab.cs.berkeley.edu/jenkins/job/spark-branch-2.2-test-maven-hadoop-2.7/47/ and found it took several seconds to create Kafka internal topic `__consumer_offsets`. As Kafka creates this topic lazily, the topic creation happens in the first test `deserialization of initial offset with Spark 2.1.0` and causes it timeout. This PR changes `offsets.topic.num.partitions` from the default value 50 to 1 to make creating `__consumer_offsets` (50 partitions -> 1 partition) much faster. ## How was this patch tested? Jenkins Author: Shixiong Zhu <shixiong@databricks.com> Closes apache#17863 from zsxwing/fix-kafka-flaky-test.
1 parent 41439fd commit bd57882

File tree

1 file changed

+1
-0
lines changed

1 file changed

+1
-0
lines changed

external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -292,6 +292,7 @@ class KafkaTestUtils(withBrokerProps: Map[String, Object] = Map.empty) extends L
292292
props.put("log.flush.interval.messages", "1")
293293
props.put("replica.socket.timeout.ms", "1500")
294294
props.put("delete.topic.enable", "true")
295+
props.put("offsets.topic.num.partitions", "1")
295296
props.putAll(withBrokerProps.asJava)
296297
props
297298
}

0 commit comments

Comments
 (0)