Skip to content

Commit 0ce697e

Browse files
zhaohehuhupan3793
zhaohehuhu
authored and
pan3793
committed
[KYUUBI apache#5654] Introduce new config to forcibly rewrite pod name
### _Why are the changes needed?_ close apache#5654 This pr is to introduce a config for rewriting pod name, which can help users forcibly rewrite driver or executor pod name prefix. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No. Closes apache#5672 from zhaohehuhu/dev-1111. Closes apache#5654 6a450e4 [hezhao2] introduce new config to force rewrite pod name Authored-by: hezhao2 <hezhao2@cisco.com> Signed-off-by: Cheng Pan <chengpan@apache.org>
1 parent 4be584c commit 0ce697e

File tree

5 files changed

+85
-26
lines changed

5 files changed

+85
-26
lines changed

‎docs/configuration/settings.md‎

Lines changed: 16 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -309,20 +309,22 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
309309

310310
### Kubernetes
311311

312-
| Key | Default | Meaning | Type | Since |
313-
|-----------------------------------------------------|-------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
314-
| kyuubi.kubernetes.authenticate.caCertFile | &lt;undefined&gt; | Path to the CA cert file for connecting to the Kubernetes API server over TLS from the kyuubi. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
315-
| kyuubi.kubernetes.authenticate.clientCertFile | &lt;undefined&gt; | Path to the client cert file for connecting to the Kubernetes API server over TLS from the kyuubi. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
316-
| kyuubi.kubernetes.authenticate.clientKeyFile | &lt;undefined&gt; | Path to the client key file for connecting to the Kubernetes API server over TLS from the kyuubi. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
317-
| kyuubi.kubernetes.authenticate.oauthToken | &lt;undefined&gt; | The OAuth token to use when authenticating against the Kubernetes API server. Note that unlike, the other authentication options, this must be the exact string value of the token to use for the authentication. | string | 1.7.0 |
318-
| kyuubi.kubernetes.authenticate.oauthTokenFile | &lt;undefined&gt; | Path to the file containing the OAuth token to use when authenticating against the Kubernetes API server. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
319-
| kyuubi.kubernetes.context | &lt;undefined&gt; | The desired context from your kubernetes config file used to configure the K8s client for interacting with the cluster. | string | 1.6.0 |
320-
| kyuubi.kubernetes.context.allow.list || The allowed kubernetes context list, if it is empty, there is no kubernetes context limitation. | set | 1.8.0 |
321-
| kyuubi.kubernetes.master.address | &lt;undefined&gt; | The internal Kubernetes master (API server) address to be used for kyuubi. | string | 1.7.0 |
322-
| kyuubi.kubernetes.namespace | default | The namespace that will be used for running the kyuubi pods and find engines. | string | 1.7.0 |
323-
| kyuubi.kubernetes.namespace.allow.list || The allowed kubernetes namespace list, if it is empty, there is no kubernetes namespace limitation. | set | 1.8.0 |
324-
| kyuubi.kubernetes.terminatedApplicationRetainPeriod | PT5M | The period for which the Kyuubi server retains application information after the application terminates. | duration | 1.7.1 |
325-
| kyuubi.kubernetes.trust.certificates | false | If set to true then client can submit to kubernetes cluster only with token | boolean | 1.7.0 |
312+
| Key | Default | Meaning | Type | Since |
313+
|----------------------------------------------------------------------|-------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
314+
| kyuubi.kubernetes.authenticate.caCertFile | &lt;undefined&gt; | Path to the CA cert file for connecting to the Kubernetes API server over TLS from the kyuubi. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
315+
| kyuubi.kubernetes.authenticate.clientCertFile | &lt;undefined&gt; | Path to the client cert file for connecting to the Kubernetes API server over TLS from the kyuubi. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
316+
| kyuubi.kubernetes.authenticate.clientKeyFile | &lt;undefined&gt; | Path to the client key file for connecting to the Kubernetes API server over TLS from the kyuubi. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
317+
| kyuubi.kubernetes.authenticate.oauthToken | &lt;undefined&gt; | The OAuth token to use when authenticating against the Kubernetes API server. Note that unlike, the other authentication options, this must be the exact string value of the token to use for the authentication. | string | 1.7.0 |
318+
| kyuubi.kubernetes.authenticate.oauthTokenFile | &lt;undefined&gt; | Path to the file containing the OAuth token to use when authenticating against the Kubernetes API server. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
319+
| kyuubi.kubernetes.context | &lt;undefined&gt; | The desired context from your kubernetes config file used to configure the K8s client for interacting with the cluster. | string | 1.6.0 |
320+
| kyuubi.kubernetes.context.allow.list || The allowed kubernetes context list, if it is empty, there is no kubernetes context limitation. | set | 1.8.0 |
321+
| kyuubi.kubernetes.master.address | &lt;undefined&gt; | The internal Kubernetes master (API server) address to be used for kyuubi. | string | 1.7.0 |
322+
| kyuubi.kubernetes.namespace | default | The namespace that will be used for running the kyuubi pods and find engines. | string | 1.7.0 |
323+
| kyuubi.kubernetes.namespace.allow.list || The allowed kubernetes namespace list, if it is empty, there is no kubernetes namespace limitation. | set | 1.8.0 |
324+
| kyuubi.kubernetes.spark.forciblyRewriteDriverPodName.enabled | false | Whether to forcibly rewrite Spark driver pod name with 'kyuubi-<uuid>-driver'. If disabled, Kyuubi will try to preserve the application name while satisfying K8s' pod name policy, but some vendors may have stricter pod name policies, thus the generated name may become illegal. | boolean | 1.8.1 |
325+
| kyuubi.kubernetes.spark.forciblyRewriteExecutorPodNamePrefix.enabled | false | Whether to forcibly rewrite Spark executor pod name prefix with 'kyuubi-<uuid>'. If disabled, Kyuubi will try to preserve the application name while satisfying K8s' pod name policy, but some vendors may have stricter Pod name policies, thus the generated name may become illegal. | boolean | 1.8.1 |
326+
| kyuubi.kubernetes.terminatedApplicationRetainPeriod | PT5M | The period for which the Kyuubi server retains application information after the application terminates. | duration | 1.7.1 |
327+
| kyuubi.kubernetes.trust.certificates | false | If set to true then client can submit to kubernetes cluster only with token | boolean | 1.7.0 |
326328

327329
### Lineage
328330

‎kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3153,4 +3153,24 @@ object KyuubiConf {
31533153
.serverOnly
31543154
.intConf
31553155
.createOptional
3156+
3157+
val KUBERNETES_FORCIBLY_REWRITE_DRIVER_POD_NAME: ConfigEntry[Boolean] =
3158+
buildConf("kyuubi.kubernetes.spark.forciblyRewriteDriverPodName.enabled")
3159+
.doc("Whether to forcibly rewrite Spark driver pod name with 'kyuubi-<uuid>-driver'. " +
3160+
"If disabled, Kyuubi will try to preserve the application name while satisfying K8s' " +
3161+
"pod name policy, but some vendors may have stricter pod name policies, thus the " +
3162+
"generated name may become illegal.")
3163+
.version("1.8.1")
3164+
.booleanConf
3165+
.createWithDefault(false)
3166+
3167+
val KUBERNETES_FORCIBLY_REWRITE_EXEC_POD_NAME_PREFIX: ConfigEntry[Boolean] =
3168+
buildConf("kyuubi.kubernetes.spark.forciblyRewriteExecutorPodNamePrefix.enabled")
3169+
.doc("Whether to forcibly rewrite Spark executor pod name prefix with 'kyuubi-<uuid>'. " +
3170+
"If disabled, Kyuubi will try to preserve the application name while satisfying K8s' " +
3171+
"pod name policy, but some vendors may have stricter Pod name policies, thus the " +
3172+
"generated name may become illegal.")
3173+
.version("1.8.1")
3174+
.booleanConf
3175+
.createWithDefault(false)
31563176
}

‎kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ import org.apache.hadoop.security.UserGroupInformation
3030

3131
import org.apache.kyuubi._
3232
import org.apache.kyuubi.config.KyuubiConf
33+
import org.apache.kyuubi.config.KyuubiConf._
3334
import org.apache.kyuubi.engine.{ApplicationManagerInfo, KyuubiApplicationManager, ProcBuilder}
3435
import org.apache.kyuubi.engine.KubernetesApplicationOperation.{KUBERNETES_SERVICE_HOST, KUBERNETES_SERVICE_PORT}
3536
import org.apache.kyuubi.engine.ProcBuilder.KYUUBI_ENGINE_LOG_PATH_KEY
@@ -229,17 +230,28 @@ class SparkProcessBuilder(
229230
kubernetesNamespace())
230231
}
231232

233+
private val forciblyRewriteDriverPodName: Boolean =
234+
conf.get(KUBERNETES_FORCIBLY_REWRITE_DRIVER_POD_NAME)
235+
private val forciblyRewriteExecPodNamePrefix: Boolean =
236+
conf.get(KUBERNETES_FORCIBLY_REWRITE_EXEC_POD_NAME_PREFIX)
237+
232238
def appendPodNameConf(conf: Map[String, String]): Map[String, String] = {
233239
val appName = conf.getOrElse(APP_KEY, "spark")
234240
val map = mutable.Map.newBuilder[String, String]
235241
if (clusterManager().exists(cm => cm.toLowerCase(Locale.ROOT).startsWith("k8s"))) {
236242
if (!conf.contains(KUBERNETES_EXECUTOR_POD_NAME_PREFIX)) {
237-
val prefix = KubernetesUtils.generateExecutorPodNamePrefix(appName, engineRefId)
243+
val prefix = KubernetesUtils.generateExecutorPodNamePrefix(
244+
appName,
245+
engineRefId,
246+
forciblyRewriteExecPodNamePrefix)
238247
map += (KUBERNETES_EXECUTOR_POD_NAME_PREFIX -> prefix)
239248
}
240249
if (deployMode().exists(_.toLowerCase(Locale.ROOT) == "cluster")) {
241250
if (!conf.contains(KUBERNETES_DRIVER_POD_NAME)) {
242-
val name = KubernetesUtils.generateDriverPodName(appName, engineRefId)
251+
val name = KubernetesUtils.generateDriverPodName(
252+
appName,
253+
engineRefId,
254+
forciblyRewriteDriverPodName)
243255
map += (KUBERNETES_DRIVER_POD_NAME -> name)
244256
}
245257
}

‎kyuubi-server/src/main/scala/org/apache/kyuubi/util/KubernetesUtils.scala‎

Lines changed: 15 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -130,21 +130,27 @@ object KubernetesUtils extends Logging {
130130
.replaceAll("^[0-9]", "x")
131131
}
132132

133-
def generateDriverPodName(appName: String, engineRefId: String): String = {
134-
val resolvedResourceName = s"kyuubi-${getResourceNamePrefix(appName, engineRefId)}-driver"
135-
if (resolvedResourceName.length <= DRIVER_POD_NAME_MAX_LENGTH) {
136-
resolvedResourceName
137-
} else {
133+
def generateDriverPodName(
134+
appName: String,
135+
engineRefId: String,
136+
forciblyRewrite: Boolean): String = {
137+
lazy val resolvedResourceName = s"kyuubi-${getResourceNamePrefix(appName, engineRefId)}-driver"
138+
if (forciblyRewrite || resolvedResourceName.length > DRIVER_POD_NAME_MAX_LENGTH) {
138139
s"kyuubi-$engineRefId-driver"
140+
} else {
141+
resolvedResourceName
139142
}
140143
}
141144

142-
def generateExecutorPodNamePrefix(appName: String, engineRefId: String): String = {
145+
def generateExecutorPodNamePrefix(
146+
appName: String,
147+
engineRefId: String,
148+
forciblyRewrite: Boolean): String = {
143149
val resolvedResourceName = s"kyuubi-${getResourceNamePrefix(appName, engineRefId)}"
144-
if (resolvedResourceName.length <= EXECUTOR_POD_NAME_PREFIX_MAX_LENGTH) {
145-
resolvedResourceName
146-
} else {
150+
if (forciblyRewrite || resolvedResourceName.length > EXECUTOR_POD_NAME_PREFIX_MAX_LENGTH) {
147151
s"kyuubi-$engineRefId"
152+
} else {
153+
resolvedResourceName
148154
}
149155
}
150156
}

‎kyuubi-server/src/test/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilderSuite.scala‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ import org.scalatestplus.mockito.MockitoSugar
2828

2929
import org.apache.kyuubi._
3030
import org.apache.kyuubi.config.KyuubiConf
31-
import org.apache.kyuubi.config.KyuubiConf.{ENGINE_LOG_TIMEOUT, ENGINE_SPARK_MAIN_RESOURCE}
31+
import org.apache.kyuubi.config.KyuubiConf.{ENGINE_LOG_TIMEOUT, ENGINE_SPARK_MAIN_RESOURCE, KUBERNETES_FORCIBLY_REWRITE_DRIVER_POD_NAME, KUBERNETES_FORCIBLY_REWRITE_EXEC_POD_NAME_PREFIX}
3232
import org.apache.kyuubi.engine.ProcBuilder.KYUUBI_ENGINE_LOG_PATH_KEY
3333
import org.apache.kyuubi.engine.spark.SparkProcessBuilder._
3434
import org.apache.kyuubi.ha.HighAvailabilityConf
@@ -336,6 +336,15 @@ class SparkProcessBuilderSuite extends KerberizedTestHelper with MockitoSugar {
336336
val conf4 = Map(APP_KEY -> chineseAppName)
337337
val driverPodName4 = processBuilder.appendPodNameConf(conf4).get(KUBERNETES_DRIVER_POD_NAME)
338338
assert(driverPodName4 === Some(s"kyuubi-test-$engineRefId-driver"))
339+
val newProcessBuilder = new SparkProcessBuilder(
340+
"kyuubi",
341+
conf.set(MASTER_KEY, "k8s://internal").set(DEPLOY_MODE_KEY, "cluster").set(
342+
KUBERNETES_FORCIBLY_REWRITE_DRIVER_POD_NAME,
343+
true),
344+
engineRefId)
345+
val conf5 = Map(APP_KEY -> "test-forcibly-rewrite-app")
346+
val driverPodName5 = newProcessBuilder.appendPodNameConf(conf5).get(KUBERNETES_DRIVER_POD_NAME)
347+
assert(driverPodName5 === Some(s"kyuubi-$engineRefId-driver"))
339348
}
340349

341350
test("[KYUUBI #5165] Test SparkProcessBuilder#appendExecutorPodPrefix") {
@@ -363,6 +372,16 @@ class SparkProcessBuilderSuite extends KerberizedTestHelper with MockitoSugar {
363372
val execPodNamePrefix3 = processBuilder
364373
.appendPodNameConf(conf3).get(KUBERNETES_EXECUTOR_POD_NAME_PREFIX)
365374
assert(execPodNamePrefix3 === Some(s"kyuubi-$engineRefId"))
375+
val newProcessBuilder = new SparkProcessBuilder(
376+
"kyuubi",
377+
conf.set(MASTER_KEY, "k8s://internal").set(DEPLOY_MODE_KEY, "cluster").set(
378+
KUBERNETES_FORCIBLY_REWRITE_EXEC_POD_NAME_PREFIX,
379+
true),
380+
engineRefId)
381+
val conf5 = Map(APP_KEY -> "test-forcibly-rewrite-app")
382+
val execPodNamePrefix4 = newProcessBuilder
383+
.appendPodNameConf(conf5).get(KUBERNETES_EXECUTOR_POD_NAME_PREFIX)
384+
assert(execPodNamePrefix4 === Some(s"kyuubi-$engineRefId"))
366385
}
367386

368387
test("extract spark core scala version") {

0 commit comments

Comments
 (0)