[SPARK-12425][STREAMING] DStream union optimisation #10382

gpoulin · 2015-12-18T16:14:53Z

Use PartitionerAwareUnionRDD when possbile for optimizing shuffling and
preserving the partitioner.

zsxwing · 2015-12-18T17:51:17Z

Could you create a JIRA and add the JIRA number to the title?

zsxwing · 2015-12-18T20:51:12Z

Jenkins, test this please

zsxwing · 2015-12-18T20:57:38Z

streaming/src/main/scala/org/apache/spark/streaming/dstream/UnionDStream.scala

@@ -46,7 +45,11 @@ class UnionDStream[T: ClassTag](parents: Array[DStream[T]])
        s" time $validTime")
    }
    if (rdds.size > 0) {
-      Some(new UnionRDD(ssc.sc, rdds))
+      if(rdds.forall(_.partitioner.isDefined) && rdds.flatMap(_.partitioner).toSet.size == 1) {


You can use Some(ssc.sc.union(rdds)) directly here

SparkQA · 2015-12-18T21:03:49Z

Test build #48025 has finished for PR 10382 at commit 839d9ee.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

gpoulin · 2015-12-22T20:29:02Z

@zsxwing based on your comment, I took the liberty to deduplicate the logic to determine if UnionRDD or a PartitionerAwareUnionRDD

zsxwing · 2015-12-22T21:10:51Z

retest this please

SparkQA · 2015-12-22T22:36:35Z

Test build #48214 has finished for PR 10382 at commit acf6f4e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

zsxwing · 2015-12-22T22:39:02Z

retest this please

zsxwing · 2015-12-23T00:13:16Z

ping @davies to take a look

SparkQA · 2015-12-23T00:22:07Z

Test build #48220 has finished for PR 10382 at commit acf6f4e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tdas · 2015-12-23T00:49:46Z

Would be nice to add a unit test in the WindowOperationsSuite for this. To make sure that windowed RDDs always have the partitioner attached.

zsxwing · 2016-01-05T01:01:56Z

@gpoulin could you resolve the conflicts and add a unit test for window as per @tdas's suggestion? Thanks!

davies · 2016-03-17T04:54:37Z

LGTM, could you fix the conflict?

srowen · 2016-03-21T14:55:15Z

@gpoulin are you able to follow up on this or should I take it over?

gpoulin · 2016-03-21T14:59:45Z

Sorry, for the delay, I was quite busy lately. I should have sometime over easter to look at this.

srowen · 2016-04-01T23:14:09Z

@gpoulin I'll push this through if you'll rebase, or can just update it separately myself

Use PartitionerAwareUnionRDD when possbile for optimizing shuffling and preserving the partitioner.

Deduplicate logic to determine if `PartitionerAwareUnionRDD` should be used instead of `UnionRDD`.

srowen · 2016-04-04T00:45:50Z

Jenkins retest this please

SparkQA · 2016-04-04T02:35:49Z

Test build #54817 has finished for PR 10382 at commit 3bb5ea3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2016-04-04T16:06:33Z

Jenkins retest this please

SparkQA · 2016-04-04T17:39:56Z

Test build #54861 has finished for PR 10382 at commit 3bb5ea3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2016-04-04T18:29:03Z

Jenkins retest this please

SparkQA · 2016-04-04T20:55:55Z

Test build #54871 has finished for PR 10382 at commit 3bb5ea3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2016-04-05T01:55:00Z

Merged to master

gpoulin changed the title ~~DStream union optimisation~~ [STREAMING] DStream union optimisation Dec 18, 2015

gpoulin changed the title ~~[STREAMING] DStream union optimisation~~ [SPARK-12425][STREAMING] DStream union optimisation Dec 18, 2015

zsxwing reviewed Dec 18, 2015
View reviewed changes

gpoulin force-pushed the dstream_union_optimisation branch from acf6f4e to be31680 Compare April 3, 2016 16:28

gpoulin added 2 commits April 3, 2016 12:31

DStream union optimisation

9f5149f

Use PartitionerAwareUnionRDD when possbile for optimizing shuffling and preserving the partitioner.

Simplify RDD union

3bb5ea3

Deduplicate logic to determine if `PartitionerAwareUnionRDD` should be used instead of `UnionRDD`.

gpoulin force-pushed the dstream_union_optimisation branch from be31680 to 3bb5ea3 Compare April 3, 2016 16:33

asfgit closed this in 7201f03 Apr 5, 2016

zzcclp pushed a commit to zzcclp/spark that referenced this pull request Apr 6, 2016

[EXT][SPARK-12425][STREAMING] DStream union optimisation apache#10382

231a0d8

[SPARK-12425][STREAMING] DStream union optimisation #10382

[SPARK-12425][STREAMING] DStream union optimisation #10382

Uh oh!

Conversation

gpoulin commented Dec 18, 2015

Uh oh!

zsxwing commented Dec 18, 2015

Uh oh!

zsxwing commented Dec 18, 2015

Uh oh!

zsxwing Dec 18, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 18, 2015

Uh oh!

gpoulin commented Dec 22, 2015

Uh oh!

zsxwing commented Dec 22, 2015

Uh oh!

SparkQA commented Dec 22, 2015

Uh oh!

zsxwing commented Dec 22, 2015

Uh oh!

zsxwing commented Dec 23, 2015

Uh oh!

SparkQA commented Dec 23, 2015

Uh oh!

tdas commented Dec 23, 2015

Uh oh!

zsxwing commented Jan 5, 2016

Uh oh!

davies commented Mar 17, 2016

Uh oh!

srowen commented Mar 21, 2016

Uh oh!

gpoulin commented Mar 21, 2016

Uh oh!

srowen commented Apr 1, 2016

Uh oh!

srowen commented Apr 4, 2016

Uh oh!

SparkQA commented Apr 4, 2016

Uh oh!

srowen commented Apr 4, 2016

Uh oh!

SparkQA commented Apr 4, 2016

Uh oh!

srowen commented Apr 4, 2016

Uh oh!

SparkQA commented Apr 4, 2016

Uh oh!

srowen commented Apr 5, 2016

Uh oh!

Uh oh!