Skip to content

Commit 4f1ae3a

Browse files
BounkongKMarcelo Vanzin
authored and
Marcelo Vanzin
committed
[SPARK-23941][MESOS] Mesos task failed on specific spark app name
## What changes were proposed in this pull request? Shell escaped the name passed to spark-submit and change how conf attributes are shell escaped. ## How was this patch tested? This test has been tested manually with Hive-on-spark with mesos or with the use case described in the issue with the sparkPi application with a custom name which contains illegal shell characters. With this PR, hive-on-spark on mesos works like a charm with hive 3.0.0-SNAPSHOT. I state that this contribution is my original work and that I license the work to the project under the project’s open source license Author: Bounkong Khamphousone <bounkong.khamphousone@ebiznext.com> Closes #21014 from tiboun/fix/SPARK-23941. (cherry picked from commit 6782359) Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
1 parent e77d62a commit 4f1ae3a

File tree

1 file changed

+2
-2
lines changed

1 file changed

+2
-2
lines changed

resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -485,9 +485,9 @@ private[spark] class MesosClusterScheduler(
485485
.filter { case (key, _) => !replicatedOptionsBlacklist.contains(key) }
486486
.toMap
487487
(defaultConf ++ driverConf).foreach { case (key, value) =>
488-
options ++= Seq("--conf", s""""$key=${shellEscape(value)}"""".stripMargin) }
488+
options ++= Seq("--conf", s"${key}=${value}") }
489489

490-
options
490+
options.map(shellEscape)
491491
}
492492

493493
/**

0 commit comments

Comments
 (0)