[SPARK-2823][GraphX]fix GraphX EdgeRDD zipPartitions

[luluorta]

If the users set “spark.default.parallelism” and the value is different with the EdgeRDD partition number, GraphX jobs will throw:
java.lang.IllegalArgumentException: Can't zip RDDs with unequal numbers of partitions

[AmplabJenkins]

Can one of the admins verify this patch?

[ankurdave]

ok to test

[SparkQA]

QA tests have started for PR 1763. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17863/consoleFull

[SparkQA]

QA results for PR 1763:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17863/consoleFull

[ankurdave]

Sorry for the delay on this. It would be great if the PR also added a unit test to reproduce the bug. I can add that if you don't have time.

[ankurdave]

Thanks! I added a test, verified that it failed before and succeeds now, and merged this into master, branch-1.1, and branch-1.0.

[Earne]

@ankurdave Ithink you miss this PR when you [Extract interfaces for EdgeRDD and VertexRDD[(https://github.com//pull/2530).

SPARK-2823 was reopened due to this.
Can we just hava a param numPartitions in EdgeRDD.scala like what VertexRDD#L313 did?
Is coalesce necessary in GraphLoader#L70? RDD after coalesce(numEdgePartitions) may not have partitions.length == numEdgePartitions

[luluorta]

Thanks, @Earne

Actually we already had a method to customize the partition number of EdgeRDD by using Graph.partitionBy Graph.scala#L136.

I guess the better name for the param of coalesce(numEdgePartitions) is maxEdgePartitions, cause it is used for making sure the generated EdgeRDD with no more than maxEdgePartitions partitions.