Skip to content

Commit 17c162f

Browse files
JerryLeadankurdave
JerryLead
authored andcommitted
[SPARK-4672][GraphX]Non-transient PartitionsRDDs will lead to StackOverflow error
The related JIRA is https://issues.apache.org/jira/browse/SPARK-4672 In a nutshell, if `val partitionsRDD` in EdgeRDDImpl and VertexRDDImpl are non-transient, the serialization chain can become very long in iterative algorithms and finally lead to the StackOverflow error. More details and explanation can be found in the JIRA. Author: JerryLead <JerryLead@163.com> Author: Lijie Xu <csxulijie@gmail.com> Closes #3544 from JerryLead/my_graphX and squashes the following commits: 628f33c [JerryLead] set PartitionsRDD to be transient in EdgeRDDImpl and VertexRDDImpl c0169da [JerryLead] Merge branch 'master' of https://github.com/apache/spark 52799e3 [Lijie Xu] Merge pull request #1 from apache/master
1 parent fc0a147 commit 17c162f

File tree

2 files changed

+2
-2
lines changed

2 files changed

+2
-2
lines changed

graphx/src/main/scala/org/apache/spark/graphx/impl/EdgeRDDImpl.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ import org.apache.spark.storage.StorageLevel
2626
import org.apache.spark.graphx._
2727

2828
class EdgeRDDImpl[ED: ClassTag, VD: ClassTag] private[graphx] (
29-
override val partitionsRDD: RDD[(PartitionID, EdgePartition[ED, VD])],
29+
@transient override val partitionsRDD: RDD[(PartitionID, EdgePartition[ED, VD])],
3030
val targetStorageLevel: StorageLevel = StorageLevel.MEMORY_ONLY)
3131
extends EdgeRDD[ED](partitionsRDD.context, List(new OneToOneDependency(partitionsRDD))) {
3232

graphx/src/main/scala/org/apache/spark/graphx/impl/VertexRDDImpl.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ import org.apache.spark.storage.StorageLevel
2727
import org.apache.spark.graphx._
2828

2929
class VertexRDDImpl[VD] private[graphx] (
30-
val partitionsRDD: RDD[ShippableVertexPartition[VD]],
30+
@transient val partitionsRDD: RDD[ShippableVertexPartition[VD]],
3131
val targetStorageLevel: StorageLevel = StorageLevel.MEMORY_ONLY)
3232
(implicit override protected val vdTag: ClassTag[VD])
3333
extends VertexRDD[VD](partitionsRDD.context, List(new OneToOneDependency(partitionsRDD))) {

0 commit comments

Comments
 (0)