Skip to content

Commit ada310a

Browse files
andrewor14pwendell
authored andcommitted
[Hot Fix #42] Persisted RDD disappears on storage page if re-used
If a previously persisted RDD is re-used, its information disappears from the Storage page. This is because the tasks associated with re-using the RDD do not report the RDD's blocks as updated (which is correct). On stage submit, however, we overwrite any existing information regarding that RDD with a fresh one, whether or not the information for the RDD already exists. Author: Andrew Or <andrewor14@gmail.com> Closes #281 from andrewor14/ui-storage-fix and squashes the following commits: 408585a [Andrew Or] Fix storage UI bug
1 parent 94fe7fd commit ada310a

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

core/src/main/scala/org/apache/spark/ui/storage/BlockManagerUI.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -84,7 +84,7 @@ private[ui] class BlockManagerListener(storageStatusListener: StorageStatusListe
8484

8585
override def onStageSubmitted(stageSubmitted: SparkListenerStageSubmitted) = synchronized {
8686
val rddInfo = stageSubmitted.stageInfo.rddInfo
87-
_rddInfoMap(rddInfo.id) = rddInfo
87+
_rddInfoMap.getOrElseUpdate(rddInfo.id, rddInfo)
8888
}
8989

9090
override def onStageCompleted(stageCompleted: SparkListenerStageCompleted) = synchronized {

0 commit comments

Comments
 (0)