Skip to content

Commit 809d4a1

Browse files
turboFeiturboFei
committed
[KYUUBI #5711] [K8S] Support to build application state and error from pod container
# 🔍 Description This pr is to support to build the application state from pod container instead of pod. The use case is that: For spark on k8s, the pod contains two containers. - one is the spark driver main container - the another one is a sidecar container If the spark driver main container completed but the sidecar container not, the batch application state is always RUNNING as the pod state is `NotReady`. It is not expected. So in this pr, I want to introduce a new option config to support to build application state from the pod container. ## Issue References 🔗 This pull request fixes # ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5711 from turboFei/k8s_pod_state. Closes #5711 9e0575b [fwang12] save eaec5d2 [fwang12] comment 61ae5f7 [fwang12] comment 1eac84f [fwang12] save bca19be [fwang12] nit 38115b7 [fwang12] [K8S] Support to build application state and error from pod container Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> (cherry picked from commit 765cdaa) Signed-off-by: fwang12 <fwang12@ebay.com>
1 parent 3117046 commit 809d4a1

File tree

4 files changed

+134
-33
lines changed

4 files changed

+134
-33
lines changed

‎docs/configuration/settings.md‎

Lines changed: 19 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -309,23 +309,25 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
309309

310310
### Kubernetes
311311

312-
| Key | Default | Meaning | Type | Since |
313-
|----------------------------------------------------------------------|-------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
314-
| kyuubi.kubernetes.authenticate.caCertFile | &lt;undefined&gt; | Path to the CA cert file for connecting to the Kubernetes API server over TLS from the kyuubi. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
315-
| kyuubi.kubernetes.authenticate.clientCertFile | &lt;undefined&gt; | Path to the client cert file for connecting to the Kubernetes API server over TLS from the kyuubi. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
316-
| kyuubi.kubernetes.authenticate.clientKeyFile | &lt;undefined&gt; | Path to the client key file for connecting to the Kubernetes API server over TLS from the kyuubi. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
317-
| kyuubi.kubernetes.authenticate.oauthToken | &lt;undefined&gt; | The OAuth token to use when authenticating against the Kubernetes API server. Note that unlike, the other authentication options, this must be the exact string value of the token to use for the authentication. | string | 1.7.0 |
318-
| kyuubi.kubernetes.authenticate.oauthTokenFile | &lt;undefined&gt; | Path to the file containing the OAuth token to use when authenticating against the Kubernetes API server. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
319-
| kyuubi.kubernetes.context | &lt;undefined&gt; | The desired context from your kubernetes config file used to configure the K8s client for interacting with the cluster. | string | 1.6.0 |
320-
| kyuubi.kubernetes.context.allow.list || The allowed kubernetes context list, if it is empty, there is no kubernetes context limitation. | set | 1.8.0 |
321-
| kyuubi.kubernetes.master.address | &lt;undefined&gt; | The internal Kubernetes master (API server) address to be used for kyuubi. | string | 1.7.0 |
322-
| kyuubi.kubernetes.namespace | default | The namespace that will be used for running the kyuubi pods and find engines. | string | 1.7.0 |
323-
| kyuubi.kubernetes.namespace.allow.list || The allowed kubernetes namespace list, if it is empty, there is no kubernetes namespace limitation. | set | 1.8.0 |
324-
| kyuubi.kubernetes.spark.deleteDriverPodOnTermination.enabled | false | If set to true then Kyuubi server will delete the spark driver pod after the application terminates for kyuubi.kubernetes.terminatedApplicationRetainPeriod. | boolean | 1.8.1 |
325-
| kyuubi.kubernetes.spark.forciblyRewriteDriverPodName.enabled | false | Whether to forcibly rewrite Spark driver pod name with 'kyuubi-<uuid>-driver'. If disabled, Kyuubi will try to preserve the application name while satisfying K8s' pod name policy, but some vendors may have stricter pod name policies, thus the generated name may become illegal. | boolean | 1.8.1 |
326-
| kyuubi.kubernetes.spark.forciblyRewriteExecutorPodNamePrefix.enabled | false | Whether to forcibly rewrite Spark executor pod name prefix with 'kyuubi-<uuid>'. If disabled, Kyuubi will try to preserve the application name while satisfying K8s' pod name policy, but some vendors may have stricter Pod name policies, thus the generated name may become illegal. | boolean | 1.8.1 |
327-
| kyuubi.kubernetes.terminatedApplicationRetainPeriod | PT5M | The period for which the Kyuubi server retains application information after the application terminates. | duration | 1.7.1 |
328-
| kyuubi.kubernetes.trust.certificates | false | If set to true then client can submit to kubernetes cluster only with token | boolean | 1.7.0 |
312+
| Key | Default | Meaning | Type | Since |
313+
|----------------------------------------------------------------------|-------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
314+
| kyuubi.kubernetes.application.state.container | spark-kubernetes-driver | The container name to retrieve the application state from. | string | 1.8.1 |
315+
| kyuubi.kubernetes.application.state.source | POD | The source to retrieve the application state from. The valid values are pod and container. If the source is container and there is container inside the pod with the name of kyuubi.kubernetes.application.state.container, the application state will be from the matched container state. Otherwise, the application state will be from the pod state. | string | 1.8.1 |
316+
| kyuubi.kubernetes.authenticate.caCertFile | &lt;undefined&gt; | Path to the CA cert file for connecting to the Kubernetes API server over TLS from the kyuubi. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
317+
| kyuubi.kubernetes.authenticate.clientCertFile | &lt;undefined&gt; | Path to the client cert file for connecting to the Kubernetes API server over TLS from the kyuubi. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
318+
| kyuubi.kubernetes.authenticate.clientKeyFile | &lt;undefined&gt; | Path to the client key file for connecting to the Kubernetes API server over TLS from the kyuubi. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
319+
| kyuubi.kubernetes.authenticate.oauthToken | &lt;undefined&gt; | The OAuth token to use when authenticating against the Kubernetes API server. Note that unlike, the other authentication options, this must be the exact string value of the token to use for the authentication. | string | 1.7.0 |
320+
| kyuubi.kubernetes.authenticate.oauthTokenFile | &lt;undefined&gt; | Path to the file containing the OAuth token to use when authenticating against the Kubernetes API server. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
321+
| kyuubi.kubernetes.context | &lt;undefined&gt; | The desired context from your kubernetes config file used to configure the K8s client for interacting with the cluster. | string | 1.6.0 |
322+
| kyuubi.kubernetes.context.allow.list || The allowed kubernetes context list, if it is empty, there is no kubernetes context limitation. | set | 1.8.0 |
323+
| kyuubi.kubernetes.master.address | &lt;undefined&gt; | The internal Kubernetes master (API server) address to be used for kyuubi. | string | 1.7.0 |
324+
| kyuubi.kubernetes.namespace | default | The namespace that will be used for running the kyuubi pods and find engines. | string | 1.7.0 |
325+
| kyuubi.kubernetes.namespace.allow.list || The allowed kubernetes namespace list, if it is empty, there is no kubernetes namespace limitation. | set | 1.8.0 |
326+
| kyuubi.kubernetes.spark.deleteDriverPodOnTermination.enabled | false | If set to true then Kyuubi server will delete the spark driver pod after the application terminates for kyuubi.kubernetes.terminatedApplicationRetainPeriod. | boolean | 1.8.1 |
327+
| kyuubi.kubernetes.spark.forciblyRewriteDriverPodName.enabled | false | Whether to forcibly rewrite Spark driver pod name with 'kyuubi-<uuid>-driver'. If disabled, Kyuubi will try to preserve the application name while satisfying K8s' pod name policy, but some vendors may have stricter pod name policies, thus the generated name may become illegal. | boolean | 1.8.1 |
328+
| kyuubi.kubernetes.spark.forciblyRewriteExecutorPodNamePrefix.enabled | false | Whether to forcibly rewrite Spark executor pod name prefix with 'kyuubi-<uuid>'. If disabled, Kyuubi will try to preserve the application name while satisfying K8s' pod name policy, but some vendors may have stricter Pod name policies, thus the generated name may become illegal. | boolean | 1.8.1 |
329+
| kyuubi.kubernetes.terminatedApplicationRetainPeriod | PT5M | The period for which the Kyuubi server retains application information after the application terminates. | duration | 1.7.1 |
330+
| kyuubi.kubernetes.trust.certificates | false | If set to true then client can submit to kubernetes cluster only with token | boolean | 1.7.0 |
329331

330332
### Lineage
331333

‎kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1239,6 +1239,30 @@ object KyuubiConf {
12391239
.booleanConf
12401240
.createWithDefault(false)
12411241

1242+
val KUBERNETES_APPLICATION_STATE_CONTAINER: ConfigEntry[String] =
1243+
buildConf("kyuubi.kubernetes.application.state.container")
1244+
.doc("The container name to retrieve the application state from.")
1245+
.version("1.8.1")
1246+
.stringConf
1247+
.createWithDefault("spark-kubernetes-driver")
1248+
1249+
val KUBERNETES_APPLICATION_STATE_SOURCE: ConfigEntry[String] =
1250+
buildConf("kyuubi.kubernetes.application.state.source")
1251+
.doc("The source to retrieve the application state from. The valid values are " +
1252+
"pod and container. If the source is container and there is container inside the pod " +
1253+
s"with the name of ${KUBERNETES_APPLICATION_STATE_CONTAINER.key}, the application state " +
1254+
s"will be from the matched container state. " +
1255+
s"Otherwise, the application state will be from the pod state.")
1256+
.version("1.8.1")
1257+
.stringConf
1258+
.checkValues(KubernetesApplicationStateSource)
1259+
.createWithDefault(KubernetesApplicationStateSource.POD.toString)
1260+
1261+
object KubernetesApplicationStateSource extends Enumeration {
1262+
type KubernetesApplicationStateSource = Value
1263+
val POD, CONTAINER = Value
1264+
}
1265+
12421266
// ///////////////////////////////////////////////////////////////////////////////////////////////
12431267
// SQL Engine Configuration //
12441268
// ///////////////////////////////////////////////////////////////////////////////////////////////

‎kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KubernetesApplicationAuditLogger.scala‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -20,22 +20,30 @@ package org.apache.kyuubi.engine
2020
import io.fabric8.kubernetes.api.model.Pod
2121

2222
import org.apache.kyuubi.Logging
23-
import org.apache.kyuubi.engine.KubernetesApplicationOperation.{toApplicationState, LABEL_KYUUBI_UNIQUE_KEY, SPARK_APP_ID_LABEL}
23+
import org.apache.kyuubi.config.KyuubiConf.KubernetesApplicationStateSource.KubernetesApplicationStateSource
24+
import org.apache.kyuubi.engine.KubernetesApplicationOperation.{toApplicationStateAndError, LABEL_KYUUBI_UNIQUE_KEY, SPARK_APP_ID_LABEL}
2425

2526
object KubernetesApplicationAuditLogger extends Logging {
2627
final private val AUDIT_BUFFER = new ThreadLocal[StringBuilder]() {
2728
override protected def initialValue: StringBuilder = new StringBuilder()
2829
}
2930

30-
def audit(kubernetesInfo: KubernetesInfo, pod: Pod): Unit = {
31+
def audit(
32+
kubernetesInfo: KubernetesInfo,
33+
pod: Pod,
34+
appStateSource: KubernetesApplicationStateSource,
35+
appStateContainer: String): Unit = {
3136
val sb = AUDIT_BUFFER.get()
3237
sb.setLength(0)
3338
sb.append(s"label=${pod.getMetadata.getLabels.get(LABEL_KYUUBI_UNIQUE_KEY)}").append("\t")
3439
sb.append(s"context=${kubernetesInfo.context.orNull}").append("\t")
3540
sb.append(s"namespace=${kubernetesInfo.namespace.orNull}").append("\t")
3641
sb.append(s"pod=${pod.getMetadata.getName}").append("\t")
3742
sb.append(s"appId=${pod.getMetadata.getLabels.get(SPARK_APP_ID_LABEL)}").append("\t")
38-
sb.append(s"appState=${toApplicationState(pod.getStatus.getPhase)}")
43+
val (appState, appError) =
44+
toApplicationStateAndError(pod, appStateSource, appStateContainer)
45+
sb.append(s"appState=$appState").append("\t")
46+
sb.append(s"appError='${appError.getOrElse("")}'")
3947
info(sb.toString())
4048
}
4149
}

0 commit comments

Comments
 (0)