[SPARK-47573][K8S] Support custom driver log url #45728

EnricoMi · 2024-03-26T18:56:45Z

What changes were proposed in this pull request?

Add config spark.ui.custom.driver.log.url to Kubernetes resource manager allow setting log url templates similar to spark.ui.custom.executor.log.url.

Allows for

--conf spark.ui.custom.driver.log.url="https://my-custom.url/api/logs?applicationId={{APP_ID}}

Supports these variables:

APP_ID: The unique application id
KUBERNETES_NAMESPACE: The namespace where the executor pods run
KUBERNETES_POD_NAME: The name of the pod that contains the executor
FILE_NAME: The name of the log, which is always "log"

Adds driver pod environment variable SPARK_DRIVER_POD_NAME.

Why are the changes needed?

Running Spark on Kubernetes requires persisting the logs elsewhere. Having the Spark UI link to those logs is very useful. This should be possible for driver logs similar to executor logs.

Does this PR introduce any user-facing change?

Spark UI provides links to logs when run on Kubernetes.

How was this patch tested?

Unit test and manually tested on minikube K8S cluster.

Was this patch authored or co-authored using generative AI tooling?

No

EnricoMi · 2024-03-26T19:02:32Z

CC @dongjoon-hyun

EnricoMi · 2024-04-20T16:27:10Z

@dongjoon-hyun What do you think?

github-actions · 2024-07-30T00:21:08Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

dongjoon-hyun · 2024-08-10T06:04:37Z

Sorry for being late. For the live log, can we try to use Spark Driver Live Log instead?

spark.driver.log.localDir=/tmp/logs

Apache Spark 4.0.0-preview1

EnricoMi · 2024-08-14T07:25:08Z

@dongjoon-hyun thanks for the pointer! Is the live log supported by the Spark History Server? Or are they only available during runtime of the Spark app? Are they log collected? Executor logs are links to the log collections service but driver logs are not? Sounds different to the spark.ui.custom.executor.log.url approach.

EnricoMi · 2024-08-14T18:12:17Z

Looks like the Spark History Server does not contain that live log tab. How is this then an equivalent replacement for the proposed driver log url?

This PR allows for the log link for the driver in the executors tab:

dongjoon-hyun · 2024-08-14T18:17:29Z

It's not the same of course, @EnricoMi .

Looks like the Spark History Server does not contain that live log tab. How is this then an equivalent replacement for the proposed driver log url?

What I wrote is For the live log, can we try to use Spark Driver Live Log instead?. Nothing more.
Have you use Apache Spark's Driver log feature before, @EnricoMi ?

EnricoMi · 2024-08-15T05:34:24Z

Haven't seen the live log feature before.

dongjoon-hyun · 2024-08-15T06:15:17Z

Ya, maybe I made you confused.

I didn't review this PR yet.
I visited SPARK-49149 first and checked the code.
- After that, I searched the PR list and see your executor log PR.
- Then, it lead me to here (driver log PR).

The Dirver Live UI feature is only a part of the following at Spark 4.

https://issues.apache.org/jira/browse/SPARK-45869 (Revisit and Improve Spark Standalone Cluster)

So, it is not relevant to this PR technically.

EnricoMi · 2024-08-15T07:16:57Z

Thanks for clarification.

dongjoon-hyun · 2024-08-15T07:30:36Z

Let me reopen this for now because it seems to be closed by the bot.

EnricoMi · 2024-08-23T08:05:06Z

@dongjoon-hyun any thoughts on this?

EnricoMi · 2024-09-02T09:12:34Z

@dongjoon-hyun can we get this into the Spark 4.0.0 release?

dongjoon-hyun · 2024-09-12T15:59:07Z

Let me check this PR quickly since I'm currently preparing Apache Spark 4.0.0-preview2, @EnricoMi .

dongjoon-hyun · 2024-09-12T16:00:22Z

core/src/main/scala/org/apache/spark/internal/config/UI.scala

@@ -220,8 +220,20 @@ private[spark] object UI {
    .stringConf
    .createOptional

+  val CUSTOM_DRIVER_LOG_URL = ConfigBuilder("spark.ui.custom.driver.log.url")


Since this doesn't have spark.kubernetes. prefix, this is supposed to work with YARN, @EnricoMi . Are you sure?

This was just derived from spark.ui.custom.executor.log.url, where I suspect this should be similar. The property name does not mention Yarn, and documentation says that only K8s support this.

core/src/main/scala/org/apache/spark/internal/config/UI.scala

docs/configuration.md

dongjoon-hyun · 2024-11-11T20:59:48Z

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Constants.scala

@@ -63,6 +63,9 @@ object Constants {
  val UI_PORT_NAME = "spark-ui"

  // Environment Variables
+  val ENV_DRIVER_ATTRIBUTE_APP_ID = "SPARK_DRIVER_ATTRIBUTE_APP_ID"
+  val ENV_DRIVER_ATTRIBUTE_KUBERNETES_NAMESPACE = "SPARK_DRIVER_ATTRIBUTE_KUBERNETES_NAMESPACE"


Just a question: we need this K8s namespace attribute additionally only for driver logs?

dongjoon-hyun · 2024-11-11T21:01:08Z

...rnetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala

    val driverCustomEnvs = KubernetesUtils.buildEnvVars(
-      Seq(ENV_APPLICATION_ID -> conf.appId) ++ conf.environment)
+      Seq(ENV_APPLICATION_ID -> conf.appId) ++ conf.environment ++ driverAttributes)


Is this inevitable to inject both ENV_APPLICATION_ID and ENV_DRIVER_ATTRIBUTE_APP_ID? This looks like a redundancy for me.

This is the same pattern as in

spark/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala

Lines 146 to 173 in 584b6db

val attributes = if (kubernetesConf.get(UI.CUSTOM_EXECUTOR_LOG_URL).isDefined) {

Map(

ENV_EXECUTOR_ATTRIBUTE_APP_ID -> kubernetesConf.appId,

ENV_EXECUTOR_ATTRIBUTE_EXECUTOR_ID -> kubernetesConf.executorId)

} else {

Map.empty[String, String]

}

KubernetesUtils.buildEnvVars(

Seq(

ENV_DRIVER_URL -> driverUrl,

ENV_EXECUTOR_CORES -> execResources.cores.get.toString,

ENV_EXECUTOR_MEMORY -> executorMemoryString,

ENV_APPLICATION_ID -> kubernetesConf.appId,

// This is to set the SPARK_CONF_DIR to be /opt/spark/conf

ENV_SPARK_CONF_DIR -> SPARK_CONF_DIR_INTERNAL,

ENV_EXECUTOR_ID -> kubernetesConf.executorId,

ENV_RESOURCE_PROFILE_ID -> resourceProfile.id.toString)

++ attributes

++ kubernetesConf.environment

++ sparkAuthSecret

++ Seq(ENV_CLASSPATH -> kubernetesConf.get(EXECUTOR_CLASS_PATH).orNull)

++ allOpts) ++

KubernetesUtils.buildEnvVarsWithFieldRef(

Seq(

(ENV_EXECUTOR_POD_IP, "v1", "status.podIP"),

(ENV_EXECUTOR_POD_NAME, "v1", "metadata.name")

))

dongjoon-hyun · 2024-11-11T21:02:47Z

core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala

@@ -144,6 +144,11 @@ class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val rpcEnv: Rp
      ThreadUtils.newDaemonSingleThreadScheduledExecutor("cleanup-decommission-execs")
    }

+  override def getDriverLogUrls: Option[Map[String, String]] = {


Instead of here, please override this at KubernetesClusterSchedulerBackend instead of CoarseGrainedSchedulerBackend because this PR aims for K8s only.

For now, only K8s cluster manager supports this configuration.

dongjoon-hyun · 2025-01-17T22:47:58Z

Could you fix the compilation errors?

EnricoMi · 2025-01-20T20:49:16Z

@dongjoon-hyun fixed

dongjoon-hyun · 2025-03-05T18:16:13Z

core/src/main/scala/org/apache/spark/internal/config/UI.scala

+      "This configuration replaces original log urls in event log, which will be also effective " +
+      "when accessing the application on history server. The new log urls must be permanent, " +
+      "otherwise you might have dead link for executor log urls.")
+    .version("4.0.0")


Shall we use 4.1.0 since master branch is 4.1.0-SNAPSHOT?

spark/pom.xml

Line 29 in a30bdc3

<version>4.1.0-SNAPSHOT</version>

dongjoon-hyun · 2025-03-05T18:16:45Z

docs/configuration.md

+    <p/>
+    For now, only K8s cluster manager supports this configuration.
+  </td>
+  <td>4.0.0</td>


ditto. Let's use 4.1.0 since master branch is 4.1.0-SNAPSHOT.

spark/pom.xml

Line 29 in a30bdc3

<version>4.1.0-SNAPSHOT</version>

…er.log.url set

…iver Pod" This reverts commit 69820e0.

…rnetesClusterSchedulerBackend

github-actions bot added KUBERNETES WEB UI DOCS CORE labels Mar 26, 2024

EnricoMi mentioned this pull request Mar 26, 2024

[SPARK-47323][K8S] Support custom executor log urls #45464

Closed

EnricoMi force-pushed the k8s-custom-driver-log-url branch 2 times, most recently from e12ba58 to a3fe442 Compare March 27, 2024 12:15

EnricoMi force-pushed the k8s-custom-driver-log-url branch from 412e9d0 to 409ce35 Compare April 20, 2024 16:23

github-actions bot added the Stale label Jul 30, 2024

EnricoMi force-pushed the k8s-custom-driver-log-url branch from 409ce35 to b0e1cde Compare July 30, 2024 08:07

github-actions bot closed this Jul 31, 2024

dongjoon-hyun removed the Stale label Aug 15, 2024

dongjoon-hyun reopened this Aug 15, 2024

dongjoon-hyun reviewed Sep 12, 2024

View reviewed changes

core/src/main/scala/org/apache/spark/internal/config/UI.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Sep 12, 2024

View reviewed changes

docs/configuration.md Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Nov 11, 2024

View reviewed changes

dongjoon-hyun removed their assignment Nov 13, 2024

EnricoMi force-pushed the k8s-custom-driver-log-url branch from 4609e87 to a09526c Compare December 2, 2024 14:38

EnricoMi force-pushed the k8s-custom-driver-log-url branch from a09526c to 188c968 Compare January 16, 2025 14:31

EnricoMi force-pushed the k8s-custom-driver-log-url branch from 188c968 to e0872a4 Compare January 20, 2025 13:52

EnricoMi force-pushed the k8s-custom-driver-log-url branch from e0872a4 to 9215c17 Compare March 5, 2025 13:19

dongjoon-hyun reviewed Mar 5, 2025

View reviewed changes

EnricoMi and others added 16 commits April 28, 2025 06:44

Add option to set custom driver log url with pattern replacement

ae0f6f5

Support custom driver log urls

4a79b33

[SPARK-41667][K8S] Expose env var SPARK_DRIVER_POD_NAME in Driver Pod

af0c1e8

Add pod name variable

87348a8

Add option to docs

fd89780

Add unit tests

bcd461e

Fix case of 'Spark' in option description

5367325

Refer to driver in docs, not executor

d285382

Revert documentation fixes

eb80803

Inject driver log url attributes via env vars if spark.ui.custom.driv…

4c2fa5d

…er.log.url set

Revert "[SPARK-41667][K8S] Expose env var SPARK_DRIVER_POD_NAME in Dr…

a0971c3

…iver Pod" This reverts commit 69820e0.

Add some SPARK_DRIVER_ATTRIBUTE_* if CUSTOM_DRIVER_LOG_URL is set

e31336f

Amend docs

cd21fe0

Move getDriverLogUrls impl from CoarseGrainedSchedulerBackend to Kube…

06c141f

…rnetesClusterSchedulerBackend

Prefix unit tests with SPARK ticket number

28e86c4

Move spark.executor.ExecutorLogUrlHandler to spark.LogUrlHandler

aebcc62

EnricoMi force-pushed the k8s-custom-driver-log-url branch from 9215c17 to aebcc62 Compare April 28, 2025 04:45

Update since-version

a974eb6

	val attributes = if (kubernetesConf.get(UI.CUSTOM_EXECUTOR_LOG_URL).isDefined) {
	Map(
	ENV_EXECUTOR_ATTRIBUTE_APP_ID -> kubernetesConf.appId,
	ENV_EXECUTOR_ATTRIBUTE_EXECUTOR_ID -> kubernetesConf.executorId)
	} else {
	Map.empty[String, String]
	}

	KubernetesUtils.buildEnvVars(
	Seq(
	ENV_DRIVER_URL -> driverUrl,
	ENV_EXECUTOR_CORES -> execResources.cores.get.toString,
	ENV_EXECUTOR_MEMORY -> executorMemoryString,
	ENV_APPLICATION_ID -> kubernetesConf.appId,
	// This is to set the SPARK_CONF_DIR to be /opt/spark/conf
	ENV_SPARK_CONF_DIR -> SPARK_CONF_DIR_INTERNAL,
	ENV_EXECUTOR_ID -> kubernetesConf.executorId,
	ENV_RESOURCE_PROFILE_ID -> resourceProfile.id.toString)
	++ attributes
	++ kubernetesConf.environment
	++ sparkAuthSecret
	++ Seq(ENV_CLASSPATH -> kubernetesConf.get(EXECUTOR_CLASS_PATH).orNull)
	++ allOpts) ++
	KubernetesUtils.buildEnvVarsWithFieldRef(
	Seq(
	(ENV_EXECUTOR_POD_IP, "v1", "status.podIP"),
	(ENV_EXECUTOR_POD_NAME, "v1", "metadata.name")
	))

[SPARK-47573][K8S] Support custom driver log url #45728

Are you sure you want to change the base?

[SPARK-47573][K8S] Support custom driver log url #45728

Conversation

EnricoMi commented Mar 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

EnricoMi commented Mar 26, 2024

Uh oh!

EnricoMi commented Apr 20, 2024

Uh oh!

github-actions bot commented Jul 30, 2024

Uh oh!

dongjoon-hyun commented Aug 10, 2024

Uh oh!

EnricoMi commented Aug 14, 2024

Uh oh!

EnricoMi commented Aug 14, 2024

Uh oh!

dongjoon-hyun commented Aug 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

EnricoMi commented Aug 15, 2024

Uh oh!

dongjoon-hyun commented Aug 15, 2024

Uh oh!

EnricoMi commented Aug 15, 2024

Uh oh!

dongjoon-hyun commented Aug 15, 2024

Uh oh!

EnricoMi commented Aug 23, 2024

Uh oh!

EnricoMi commented Sep 2, 2024

Uh oh!

dongjoon-hyun commented Sep 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

EnricoMi Sep 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jan 17, 2025

Uh oh!

EnricoMi commented Jan 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

EnricoMi commented Mar 26, 2024 •

edited

Loading

dongjoon-hyun commented Aug 14, 2024 •

edited

Loading

dongjoon-hyun commented Sep 12, 2024 •

edited

Loading

EnricoMi Sep 13, 2024 •

edited

Loading