Skip to content

Commit

Permalink
[SPARK-33724][K8S] Add decom script as a configuration param
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

Makes the location of the decommission script used in Kubernetes for graceful shutdown configurable.

### Why are the changes needed?

Some environments don't use the Spark image builder and instead mount the decompressed Spark distro. In those envs configuring the location of the decommissioning script is required.

### Does this PR introduce _any_ user-facing change?

New configuration parameter.

### How was this patch tested?

Existing decommissioning integration test.

Closes #30694 from holdenk/SPARK-33724-allow-decommissioning-script-location-to-be-configured.

Authored-by: Holden Karau <hkarau@apple.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
holdenk authored and dongjoon-hyun committed Dec 10, 2020
1 parent 991b797 commit 1c7f5f1
Show file tree
Hide file tree
Showing 2 changed files with 9 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,14 @@ import org.apache.spark.internal.config.ConfigBuilder

private[spark] object Config extends Logging {

val DECOMMISSION_SCRIPT =
ConfigBuilder("spark.kubernetes.decommission.script")
.doc("The location of the script to use for graceful decommissioning")
.version("3.2.0")
.stringConf
.createWithDefault("/opt/decom.sh")


val KUBERNETES_CONTEXT =
ConfigBuilder("spark.kubernetes.context")
.doc("The desired context from your K8S config file used to configure the K8S " +
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -230,7 +230,7 @@ private[spark] class BasicExecutorFeatureStep(
new ContainerBuilder(containerWithLimitCores).withNewLifecycle()
.withNewPreStop()
.withNewExec()
.addToCommand("/opt/decom.sh")
.addToCommand(kubernetesConf.get(DECOMMISSION_SCRIPT))
.endExec()
.endPreStop()
.endLifecycle()
Expand Down

0 comments on commit 1c7f5f1

Please sign in to comment.