[SPARK-11517][SQL]Calc partitions in parallel for multiple partitions table

[zhichao-li]

Currently we calculate the getPartitions for each "hive partition" in sequence way, it would be faster if we can parallel this on driver side

[zhichao-li]

cc @chenghao-intel

[SparkQA]

Test build #45083 has finished for PR 9483 at commit 63dc9c0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * class ParallelUnionRDD[T: ClassTag](\n

[chenghao-intel]

cc/ @scwf @Sephiroth-Lin, not sure if you guys get time for benchmarking this with the real world cases.

[zhonghaihua]

Hi @zhichao-li ,thanks for doing this.I got a problem of scanning partitions slowly,and I apply this patch to my spark version.In my case:

• Before I apply this patch,it takes at least 3 or 4 minutes to scan partitions.
• After applying this patch,it takes only about 20 seconds at this stage.

I am happy to see it takes effect in my case.It solve my problem.And I think is it better to add conf to control whether to use this feature？

[yhuai]

Should we share the single thread pool instead of creating a thread pool for every ParallelUnionRDD?

[zhichao-li]

I don't have strong opinion on this. How about creating a shared thread pool with the same size as cpu cores ?

object ParallelUnionRDD{
val executorService = ThreadUtils.newDaemonFixedThreadPool(Runtime.getRuntime.availableProcessors(), "ParallelUnionRDD")
}

[chenghao-intel]

I don't think we have to put the fixed number of Runtime.getRuntime.availableProcessors(), probably we can simply put a fixed number says 16 or even bigger, as the bottleneck is in network / IO, not the CPU scheduling.

[zhichao-li]

@yhuai @rxin , any thoughts or concerns for this PR? It's common that one table contains tons of partitions(i.e every 15mins a partition for clicking data).

[chenghao-intel]

seems here still be the main thread, probably we even don't need to place the synchronized in the getPartitions.

[SparkQA]

Test build #51856 has finished for PR 9483 at commit 6456f12.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #51861 has finished for PR 9483 at commit db84ab9.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zhichao-li]

retest this please

[SparkQA]

Test build #51920 has finished for PR 9483 at commit db84ab9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[chenghao-intel]

private[hive] or move it into the upper level package? The same for the class ParallelUnionRDD.

[chenghao-intel]

LGTM except some minor suggestions.

[SparkQA]

Test build #52152 has finished for PR 9483 at commit fdac95b.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zhichao-li]

retest this please

[SparkQA]

Test build #52156 has finished for PR 9483 at commit fdac95b.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zhichao-li]

retest this please

[SparkQA]

Test build #52210 has finished for PR 9483 at commit fdac95b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Thanks for the pull request. I'm going through a list of pull requests to cut them down since the sheer number is breaking some of the tooling we have. Due to lack of activity on this pull request, I'm going to push a commit to close it. Feel free to reopen it or create a new one. We can also continue the discussion on the JIRA ticket.