Skip to content

Commit d208047

Browse files
Marcelo VanzinAndrew Or
Marcelo Vanzin
authored and
Andrew Or
committed
[MINOR] [CORE] Warn users who try to cache RDDs with dynamic allocation on.
Author: Marcelo Vanzin <vanzin@cloudera.com> Closes apache#5751 from vanzin/cached-rdd-warning and squashes the following commits: 554cc07 [Marcelo Vanzin] Change message. 9efb9da [Marcelo Vanzin] [minor] [core] Warn users who try to cache RDDs with dynamic allocation on.
1 parent 059c390 commit d208047

File tree

1 file changed

+5
-0
lines changed

1 file changed

+5
-0
lines changed

core/src/main/scala/org/apache/spark/SparkContext.scala

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1150,6 +1150,11 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli
11501150
* Register an RDD to be persisted in memory and/or disk storage
11511151
*/
11521152
private[spark] def persistRDD(rdd: RDD[_]) {
1153+
_executorAllocationManager.foreach { _ =>
1154+
logWarning(
1155+
s"Dynamic allocation currently does not support cached RDDs. Cached data for RDD " +
1156+
s"${rdd.id} will be lost when executors are removed.")
1157+
}
11531158
persistentRdds(rdd.id) = rdd
11541159
}
11551160

0 commit comments

Comments
 (0)