[SPARK-5731][Streaming][Test] Fix incorrect test in DirectKafkaStreamSuite

[tdas]

The test was incorrect. Instead of counting the number of records, it counted the number of partitions of RDD generated by DStream. Which is not its intention. I will be testing this patch multiple times to understand its flakiness.

PS: This was caused by my refactoring in #4384

@koeninger check it out.

[SparkQA]

Test build #603 has started for PR 4597 at commit e9a1820.

• This patch merges cleanly.

[SparkQA]

Test build #604 has started for PR 4597 at commit e9a1820.

• This patch merges cleanly.

[SparkQA]

Test build #605 has started for PR 4597 at commit e9a1820.

• This patch merges cleanly.

[SparkQA]

Test build #606 has started for PR 4597 at commit e9a1820.

• This patch merges cleanly.

[SparkQA]

Test build #27455 has started for PR 4597 at commit e9a1820.

• This patch merges cleanly.

[SparkQA]

Test build #27458 has started for PR 4597 at commit d236235.

• This patch merges cleanly.

[SparkQA]

Test build #27455 has finished for PR 4597 at commit e9a1820.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27455/
Test FAILed.

[SparkQA]

Test build #604 has finished for PR 4597 at commit e9a1820.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Params(

[SparkQA]

Test build #606 has finished for PR 4597 at commit e9a1820.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[koeninger]

LGTM

[SparkQA]

Test build #603 has finished for PR 4597 at commit e9a1820.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #605 has finished for PR 4597 at commit e9a1820.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #607 has started for PR 4597 at commit d236235.

• This patch merges cleanly.

[SparkQA]

Test build #608 has started for PR 4597 at commit d236235.

• This patch merges cleanly.

[SparkQA]

Test build #609 has started for PR 4597 at commit d236235.

• This patch merges cleanly.

[SparkQA]

Test build #27458 has finished for PR 4597 at commit d236235.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27458/
Test PASSed.

[SparkQA]

Test build #609 has finished for PR 4597 at commit d236235.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #607 has finished for PR 4597 at commit d236235.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #608 has finished for PR 4597 at commit d236235.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Seems like streamingcontext itself is a bit flaky ...

[tdas]

Why? Even the failure of StreamingContext.stop() unit test in the failed Test build #609 above was causing by the same broadcast piece failure.

[info] - stop(stopSparkContext=true) after stop(stopSparkContext=false) *** FAILED *** (65 milliseconds)
[info]   org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.io.IOException: org.apache.spark.SparkException: Failed to get broadcast_0_piece0 of broadcast_0

This is indeed weird. Anyways, the Kafka test seems pretty solid now. If its okay with you I merging this, and we can continue investigating on the broadcast problem independently.

[rxin]

Please go ahead and merge.
On Feb 14, 2015 4:29 AM, "Tathagata Das" notifications@github.com wrote:

Why? Even the failure of StreamingContext.stop() unit test in the failed
Test build #609 #609 above was
causing by the same broadcast piece failure.

[info] - stop(stopSparkContext=true) after stop(stopSparkContext=false) *** FAILED *** (65 milliseconds)
[info] org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.io.IOException: org.apache.spark.SparkException: Failed to get broadcast_0_piece0 of broadcast_0

This is indeed weird. Anyways, the Kafka test seems pretty solid now. If
its okay with you I merging this, and we can continue investigating on the
broadcast problem independently.

—
Reply to this email directly or view it on GitHub
#4597 (comment).