[SPARK-14804][Spark][Graphx] Fix Graph vertexRDD/EdgeRDD checkpoint results ClassCastException

[suyanNone]

What changes were proposed in this pull request?

The PR fixed compute chain from CheckpointRDD<-vertexRDDImp to CheckpointRDD<-partitionRDD<- vertexRDDImpl

How was this patch tested?

unit test

[suyanNone]

mistake to open, close first

[SparkQA]

Test build #56543 has finished for PR 12576 at commit f2ced74.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #56635 has finished for PR 12576 at commit a106758.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tdas]

This fix is roughly in the right direction. However there are two major concerns.

• It changes the behavior of VertexRDD.isCheckpointed() which is a public method.
• There must be new tests added to make sure this does not happen again. I dont see any new tests that checkpoints, forces the checkpoint by count/collect, and then collects again.

Ping @jkbradley

[tdas]

also, ping @rxin

[tdas]

Actually, correction. Here is a better fix. The key problem here is that the RDD internal logic for computing from checkpoints, depends on a public, override-able API. This is wrong. Either the isCheckpointed should have been marked final, or the internal isCheckpointedAndMaterialized should not have depended on isCheckpointed (see here).

So the simplest, and correct solution is probably (not tested) to change isCheckpointedAndMaterialzed.

  private[spark] def isCheckpointedAndMaterialized: Boolean = checkpointData.exists(_.isCheckpointed)

This preserves the behavior of VertexRDD.isCheckpointed() and prevents such issues related to overridden RDD.isCheckpointed() from happening ever again.

[jkbradley]

CC: @ankurdave just making you aware of this

[suyanNone]

@tdas agree, isCheckpointed better be final, in current code, isCheckpointed exposed as public is for testing?

[tdas]

Okay so can you update this PR according to the discussion, and add the necessary tests as well?

[neggert]

Not sure what's going on regarding the multiple PRs for this issue, but I cherry-picked this PR on top of 1.6.2 and it fixed the problem for me.