Skip to content

Commit a44c118

Browse files
saturday_sMarcelo Vanzin
authored andcommitted
[SPARK-19688][STREAMING] Not to read spark.yarn.credentials.file from checkpoint.
## What changes were proposed in this pull request? Reload the `spark.yarn.credentials.file` property when restarting a streaming application from checkpoint. ## How was this patch tested? Manual tested with 1.6.3 and 2.1.1. I didn't test this with master because of some compile problems, but I think it will be the same result. ## Notice This should be merged into maintenance branches too. jira: [SPARK-21008](https://issues.apache.org/jira/browse/SPARK-21008) Author: saturday_s <shi.indetail@gmail.com> Closes #18230 from saturday-shi/SPARK-21008. (cherry picked from commit e92ffe6) Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
1 parent 0ebb3b8 commit a44c118

File tree

1 file changed

+3
-0
lines changed

1 file changed

+3
-0
lines changed

streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,9 @@ class Checkpoint(ssc: StreamingContext, val checkpointTime: Time)
5555
"spark.master",
5656
"spark.yarn.keytab",
5757
"spark.yarn.principal",
58+
"spark.yarn.credentials.file",
59+
"spark.yarn.credentials.renewalTime",
60+
"spark.yarn.credentials.updateTime",
5861
"spark.ui.filters")
5962

6063
val newSparkConf = new SparkConf(loadDefaults = false).setAll(sparkConfPairs)

0 commit comments

Comments
 (0)