Skip to content

Commit 3d3dacb

Browse files
onursaticibulldozer-bot[bot]
authored andcommitted
kubernetes local should work with in process launcher (#601)
## Upstream SPARK-XXXXX ticket and PR link (if not applicable, explain) local kubernetes submission is not submitted upstream yet ## What changes were proposed in this pull request? In process launcher clobbers spark configuration with launcher port and secret overrides, resulting the local spark submit on the driver to connect to the port used by the cluster mode spark submit process. Spark drivers in k8s are hard coded to run spark-submit, so spark launcher overrides are inapplicable to them. Excluding the configurations from the config map properly signals the local backend to not try to connect to a launcher server
1 parent a7df826 commit 3d3dacb

File tree

1 file changed

+3
-1
lines changed

1 file changed

+3
-1
lines changed

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -176,7 +176,9 @@ private[spark] class Client(
176176
// Build a Config Map that will house spark conf properties in a single file for spark-submit
177177
private def buildConfigMap(configMapName: String, conf: Map[String, String]): ConfigMap = {
178178
val properties = new Properties()
179-
conf.foreach { case (k, v) =>
179+
conf
180+
.filter { case (k, _) => !k.startsWith("spark.launcher")}
181+
.foreach { case (k, v) =>
180182
properties.setProperty(k, v)
181183
}
182184
val propertiesWriter = new StringWriter()

0 commit comments

Comments
 (0)