Skip to content

Commit dccb129

Browse files
panbingkundongjoon-hyun
panbingkun
authored and
dongjoon-hyun
committed
[SPARK-50493][SPARK-37687][BUILD] Migrate kubernetes-client from 6.x to 7.x
### What changes were proposed in this pull request? The pr aims to migrate kubernetes-client from `6.x` to `7.x`, includes: - upgrade `kubernetes-client` from `6.13.2` to `7.0.1`. - switch default `HttpClient` implementation from `OkHttp` to `Vert.x`. ### Why are the changes needed? - The full release notes: https://github.com/fabric8io/kubernetes-client/releases/tag/v7.0.1 https://github.com/fabric8io/kubernetes-client/releases/tag/v7.0.0 - [Vert.x as default HttpClient implementation](https://github.com/fabric8io/kubernetes-client/blob/v7.0.0/doc/MIGRATION-v7.md#vertx-as-default-httpclient-implementation-) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#49159 from panbingkun/k8s_client_7_vert_x. Authored-by: panbingkun <panbingkun@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent a59941a commit dccb129

File tree

9 files changed

+55
-61
lines changed

9 files changed

+55
-61
lines changed

‎LICENSE-binary‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -286,6 +286,10 @@ io.netty:netty-transport-classes-kqueue
286286
io.netty:netty-transport-native-epoll
287287
io.netty:netty-transport-native-kqueue
288288
io.netty:netty-transport-native-unix-common
289+
io.vertx:vertx-auth-common
290+
io.vertx:vertx-core
291+
io.vertx:vertx-web-client
292+
io.vertx:vertx-web-common
289293
jakarta.inject:jakarta.inject-api
290294
jakarta.validation:jakarta.validation-api
291295
javax.jdo:jdo-api

‎dev/deps/spark-deps-hadoop-3-hive-2.3‎

Lines changed: 33 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -158,31 +158,31 @@ jsr305/3.0.0//jsr305-3.0.0.jar
158158
jta/1.1//jta-1.1.jar
159159
jul-to-slf4j/2.0.16//jul-to-slf4j-2.0.16.jar
160160
kryo-shaded/4.0.2//kryo-shaded-4.0.2.jar
161-
kubernetes-client-api/6.13.4//kubernetes-client-api-6.13.4.jar
162-
kubernetes-client/6.13.4//kubernetes-client-6.13.4.jar
163-
kubernetes-httpclient-okhttp/6.13.4//kubernetes-httpclient-okhttp-6.13.4.jar
164-
kubernetes-model-admissionregistration/6.13.4//kubernetes-model-admissionregistration-6.13.4.jar
165-
kubernetes-model-apiextensions/6.13.4//kubernetes-model-apiextensions-6.13.4.jar
166-
kubernetes-model-apps/6.13.4//kubernetes-model-apps-6.13.4.jar
167-
kubernetes-model-autoscaling/6.13.4//kubernetes-model-autoscaling-6.13.4.jar
168-
kubernetes-model-batch/6.13.4//kubernetes-model-batch-6.13.4.jar
169-
kubernetes-model-certificates/6.13.4//kubernetes-model-certificates-6.13.4.jar
170-
kubernetes-model-common/6.13.4//kubernetes-model-common-6.13.4.jar
171-
kubernetes-model-coordination/6.13.4//kubernetes-model-coordination-6.13.4.jar
172-
kubernetes-model-core/6.13.4//kubernetes-model-core-6.13.4.jar
173-
kubernetes-model-discovery/6.13.4//kubernetes-model-discovery-6.13.4.jar
174-
kubernetes-model-events/6.13.4//kubernetes-model-events-6.13.4.jar
175-
kubernetes-model-extensions/6.13.4//kubernetes-model-extensions-6.13.4.jar
176-
kubernetes-model-flowcontrol/6.13.4//kubernetes-model-flowcontrol-6.13.4.jar
177-
kubernetes-model-gatewayapi/6.13.4//kubernetes-model-gatewayapi-6.13.4.jar
178-
kubernetes-model-metrics/6.13.4//kubernetes-model-metrics-6.13.4.jar
179-
kubernetes-model-networking/6.13.4//kubernetes-model-networking-6.13.4.jar
180-
kubernetes-model-node/6.13.4//kubernetes-model-node-6.13.4.jar
181-
kubernetes-model-policy/6.13.4//kubernetes-model-policy-6.13.4.jar
182-
kubernetes-model-rbac/6.13.4//kubernetes-model-rbac-6.13.4.jar
183-
kubernetes-model-resource/6.13.4//kubernetes-model-resource-6.13.4.jar
184-
kubernetes-model-scheduling/6.13.4//kubernetes-model-scheduling-6.13.4.jar
185-
kubernetes-model-storageclass/6.13.4//kubernetes-model-storageclass-6.13.4.jar
161+
kubernetes-client-api/7.0.1//kubernetes-client-api-7.0.1.jar
162+
kubernetes-client/7.0.1//kubernetes-client-7.0.1.jar
163+
kubernetes-httpclient-vertx/7.0.1//kubernetes-httpclient-vertx-7.0.1.jar
164+
kubernetes-model-admissionregistration/7.0.1//kubernetes-model-admissionregistration-7.0.1.jar
165+
kubernetes-model-apiextensions/7.0.1//kubernetes-model-apiextensions-7.0.1.jar
166+
kubernetes-model-apps/7.0.1//kubernetes-model-apps-7.0.1.jar
167+
kubernetes-model-autoscaling/7.0.1//kubernetes-model-autoscaling-7.0.1.jar
168+
kubernetes-model-batch/7.0.1//kubernetes-model-batch-7.0.1.jar
169+
kubernetes-model-certificates/7.0.1//kubernetes-model-certificates-7.0.1.jar
170+
kubernetes-model-common/7.0.1//kubernetes-model-common-7.0.1.jar
171+
kubernetes-model-coordination/7.0.1//kubernetes-model-coordination-7.0.1.jar
172+
kubernetes-model-core/7.0.1//kubernetes-model-core-7.0.1.jar
173+
kubernetes-model-discovery/7.0.1//kubernetes-model-discovery-7.0.1.jar
174+
kubernetes-model-events/7.0.1//kubernetes-model-events-7.0.1.jar
175+
kubernetes-model-extensions/7.0.1//kubernetes-model-extensions-7.0.1.jar
176+
kubernetes-model-flowcontrol/7.0.1//kubernetes-model-flowcontrol-7.0.1.jar
177+
kubernetes-model-gatewayapi/7.0.1//kubernetes-model-gatewayapi-7.0.1.jar
178+
kubernetes-model-metrics/7.0.1//kubernetes-model-metrics-7.0.1.jar
179+
kubernetes-model-networking/7.0.1//kubernetes-model-networking-7.0.1.jar
180+
kubernetes-model-node/7.0.1//kubernetes-model-node-7.0.1.jar
181+
kubernetes-model-policy/7.0.1//kubernetes-model-policy-7.0.1.jar
182+
kubernetes-model-rbac/7.0.1//kubernetes-model-rbac-7.0.1.jar
183+
kubernetes-model-resource/7.0.1//kubernetes-model-resource-7.0.1.jar
184+
kubernetes-model-scheduling/7.0.1//kubernetes-model-scheduling-7.0.1.jar
185+
kubernetes-model-storageclass/7.0.1//kubernetes-model-storageclass-7.0.1.jar
186186
lapack/3.0.3//lapack-3.0.3.jar
187187
leveldbjni-all/1.8//leveldbjni-all-1.8.jar
188188
libfb303/0.9.3//libfb303-0.9.3.jar
@@ -193,7 +193,6 @@ log4j-api/2.24.3//log4j-api-2.24.3.jar
193193
log4j-core/2.24.3//log4j-core-2.24.3.jar
194194
log4j-layout-template-json/2.24.3//log4j-layout-template-json-2.24.3.jar
195195
log4j-slf4j2-impl/2.24.3//log4j-slf4j2-impl-2.24.3.jar
196-
logging-interceptor/3.12.12//logging-interceptor-3.12.12.jar
197196
lz4-java/1.8.0//lz4-java-1.8.0.jar
198197
metrics-core/4.2.29//metrics-core-4.2.29.jar
199198
metrics-graphite/4.2.29//metrics-graphite-4.2.29.jar
@@ -203,13 +202,15 @@ metrics-jvm/4.2.29//metrics-jvm-4.2.29.jar
203202
minlog/1.3.0//minlog-1.3.0.jar
204203
netty-all/4.1.115.Final//netty-all-4.1.115.Final.jar
205204
netty-buffer/4.1.115.Final//netty-buffer-4.1.115.Final.jar
205+
netty-codec-dns/4.1.115.Final//netty-codec-dns-4.1.115.Final.jar
206206
netty-codec-http/4.1.115.Final//netty-codec-http-4.1.115.Final.jar
207207
netty-codec-http2/4.1.115.Final//netty-codec-http2-4.1.115.Final.jar
208208
netty-codec-socks/4.1.115.Final//netty-codec-socks-4.1.115.Final.jar
209209
netty-codec/4.1.115.Final//netty-codec-4.1.115.Final.jar
210210
netty-common/4.1.115.Final//netty-common-4.1.115.Final.jar
211211
netty-handler-proxy/4.1.115.Final//netty-handler-proxy-4.1.115.Final.jar
212212
netty-handler/4.1.115.Final//netty-handler-4.1.115.Final.jar
213+
netty-resolver-dns/4.1.115.Final//netty-resolver-dns-4.1.115.Final.jar
213214
netty-resolver/4.1.115.Final//netty-resolver-4.1.115.Final.jar
214215
netty-tcnative-boringssl-static/2.0.69.Final/linux-aarch_64/netty-tcnative-boringssl-static-2.0.69.Final-linux-aarch_64.jar
215216
netty-tcnative-boringssl-static/2.0.69.Final/linux-x86_64/netty-tcnative-boringssl-static-2.0.69.Final-linux-x86_64.jar
@@ -258,7 +259,7 @@ scala-parser-combinators_2.13/2.4.0//scala-parser-combinators_2.13-2.4.0.jar
258259
scala-reflect/2.13.15//scala-reflect-2.13.15.jar
259260
scala-xml_2.13/2.3.0//scala-xml_2.13-2.3.0.jar
260261
slf4j-api/2.0.16//slf4j-api-2.0.16.jar
261-
snakeyaml-engine/2.7//snakeyaml-engine-2.7.jar
262+
snakeyaml-engine/2.8//snakeyaml-engine-2.8.jar
262263
snakeyaml/2.3//snakeyaml-2.3.jar
263264
snappy-java/1.1.10.7//snappy-java-1.1.10.7.jar
264265
spire-macros_2.13/0.18.0//spire-macros_2.13-0.18.0.jar
@@ -272,11 +273,15 @@ threeten-extra/1.7.1//threeten-extra-1.7.1.jar
272273
tink/1.16.0//tink-1.16.0.jar
273274
transaction-api/1.1//transaction-api-1.1.jar
274275
univocity-parsers/2.9.1//univocity-parsers-2.9.1.jar
276+
vertx-auth-common/4.5.11//vertx-auth-common-4.5.11.jar
277+
vertx-core/4.5.11//vertx-core-4.5.11.jar
278+
vertx-web-client/4.5.11//vertx-web-client-4.5.11.jar
279+
vertx-web-common/4.5.11//vertx-web-common-4.5.11.jar
275280
wildfly-openssl/1.1.3.Final//wildfly-openssl-1.1.3.Final.jar
276281
xbean-asm9-shaded/4.26//xbean-asm9-shaded-4.26.jar
277282
xmlschema-core/2.3.1//xmlschema-core-2.3.1.jar
278283
xz/1.10//xz-1.10.jar
279-
zjsonpatch/0.3.0//zjsonpatch-0.3.0.jar
284+
zjsonpatch/7.0.1//zjsonpatch-7.0.1.jar
280285
zookeeper-jute/3.9.3//zookeeper-jute-3.9.3.jar
281286
zookeeper/3.9.3//zookeeper-3.9.3.jar
282287
zstd-jni/1.5.6-8//zstd-jni-1.5.6-8.jar

‎hadoop-cloud/pom.xml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,8 @@
3434
</description>
3535
<properties>
3636
<sbt.project.name>hadoop-cloud</sbt.project.name>
37+
<okhttp.version>3.12.12</okhttp.version>
38+
<okio.version>1.17.6</okio.version>
3739
</properties>
3840

3941
<dependencies>
@@ -191,6 +193,16 @@
191193
<version>${jetty.version}</version>
192194
<scope>${hadoop.deps.scope}</scope>
193195
</dependency>
196+
<dependency>
197+
<groupId>com.squareup.okhttp3</groupId>
198+
<artifactId>okhttp</artifactId>
199+
<version>${okhttp.version}</version>
200+
</dependency>
201+
<dependency>
202+
<groupId>com.squareup.okio</groupId>
203+
<artifactId>okio</artifactId>
204+
<version>${okio.version}</version>
205+
</dependency>
194206
</dependencies>
195207

196208
<build>

‎pom.xml‎

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -235,8 +235,7 @@
235235

236236
<!-- org.fusesource.leveldbjni will be used except on arm64 platform. -->
237237
<leveldbjni.group>org.fusesource.leveldbjni</leveldbjni.group>
238-
<kubernetes-client.version>6.13.4</kubernetes-client.version>
239-
<okio.version>1.17.6</okio.version>
238+
<kubernetes-client.version>7.0.1</kubernetes-client.version>
240239

241240
<test.java.home>${java.home}</test.java.home>
242241

@@ -2594,11 +2593,6 @@
25942593
<artifactId>javax.servlet-api</artifactId>
25952594
<version>${javaxservlet.version}</version>
25962595
</dependency>
2597-
<dependency>
2598-
<groupId>com.squareup.okio</groupId>
2599-
<artifactId>okio</artifactId>
2600-
<version>${okio.version}</version>
2601-
</dependency>
26022596
</dependencies>
26032597
</dependencyManagement>
26042598

‎resource-managers/kubernetes/core/pom.xml‎

Lines changed: 1 addition & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@
3737
<dependencies>
3838
<dependency>
3939
<groupId>io.fabric8</groupId>
40-
<artifactId>volcano-model-v1beta1</artifactId>
40+
<artifactId>volcano-model</artifactId>
4141
<version>${kubernetes-client.version}</version>
4242
</dependency>
4343
<dependency>
@@ -105,11 +105,6 @@
105105
<scope>test</scope>
106106
</dependency>
107107

108-
<dependency>
109-
<groupId>io.fabric8</groupId>
110-
<artifactId>kubernetes-httpclient-okhttp</artifactId>
111-
<version>${kubernetes-client.version}</version>
112-
</dependency>
113108
<dependency>
114109
<groupId>io.fabric8</groupId>
115110
<artifactId>kubernetes-client</artifactId>

‎resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala‎

Lines changed: 1 addition & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -24,18 +24,14 @@ import com.google.common.io.Files
2424
import io.fabric8.kubernetes.client.{ConfigBuilder, KubernetesClient, KubernetesClientBuilder}
2525
import io.fabric8.kubernetes.client.Config.KUBERNETES_REQUEST_RETRY_BACKOFFLIMIT_SYSTEM_PROPERTY
2626
import io.fabric8.kubernetes.client.Config.autoConfigure
27-
import io.fabric8.kubernetes.client.okhttp.OkHttpClientFactory
2827
import io.fabric8.kubernetes.client.utils.Utils.getSystemPropertyOrEnvVar
29-
import okhttp3.Dispatcher
30-
import okhttp3.OkHttpClient
3128

3229
import org.apache.spark.SparkConf
3330
import org.apache.spark.annotation.{DeveloperApi, Since, Stable}
3431
import org.apache.spark.deploy.k8s.Config._
3532
import org.apache.spark.internal.{Logging, MDC}
3633
import org.apache.spark.internal.LogKeys.K8S_CONTEXT
3734
import org.apache.spark.internal.config.ConfigEntry
38-
import org.apache.spark.util.ThreadUtils
3935

4036
/**
4137
* :: DeveloperApi ::
@@ -78,10 +74,6 @@ object SparkKubernetesClientFactory extends Logging {
7874
.getOption(s"$kubernetesAuthConfPrefix.$CLIENT_KEY_FILE_CONF_SUFFIX")
7975
val clientCertFile = sparkConf
8076
.getOption(s"$kubernetesAuthConfPrefix.$CLIENT_CERT_FILE_CONF_SUFFIX")
81-
// TODO(SPARK-37687): clean up direct usage of OkHttpClient, see also:
82-
// https://github.com/fabric8io/kubernetes-client/issues/3547
83-
val dispatcher = new Dispatcher(
84-
ThreadUtils.newDaemonCachedThreadPool("kubernetes-dispatcher"))
8577

8678
// Allow for specifying a context used to auto-configure from the users K8S config file
8779
val kubeContext = sparkConf.get(KUBERNETES_CONTEXT).filter(_.nonEmpty)
@@ -117,17 +109,9 @@ object SparkKubernetesClientFactory extends Logging {
117109
}.withOption(namespace) {
118110
(ns, configBuilder) => configBuilder.withNamespace(ns)
119111
}.build()
120-
val factoryWithCustomDispatcher = new OkHttpClientFactory() {
121-
override protected def additionalConfig(builder: OkHttpClient.Builder): Unit = {
122-
builder.dispatcher(dispatcher)
123-
}
124-
}
125112
logDebug("Kubernetes client config: " +
126113
new ObjectMapper().writerWithDefaultPrettyPrinter().writeValueAsString(config))
127-
new KubernetesClientBuilder()
128-
.withHttpClientFactory(factoryWithCustomDispatcher)
129-
.withConfig(config)
130-
.build()
114+
new KubernetesClientBuilder().withConfig(config).build()
131115
}
132116

133117
private implicit class OptionConfigurableConfigBuilder(val configBuilder: ConfigBuilder)

‎resource-managers/kubernetes/core/volcano/src/main/scala/org/apache/spark/deploy/k8s/features/VolcanoFeatureStep.scala‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,8 @@
1717
package org.apache.spark.deploy.k8s.features
1818

1919
import io.fabric8.kubernetes.api.model._
20+
import io.fabric8.volcano.api.model.scheduling.v1beta1.{PodGroup, PodGroupSpec}
2021
import io.fabric8.volcano.client.DefaultVolcanoClient
21-
import io.fabric8.volcano.scheduling.v1beta1.{PodGroup, PodGroupSpec}
2222

2323
import org.apache.spark.deploy.k8s.{KubernetesConf, KubernetesDriverConf, KubernetesExecutorConf, SparkPod}
2424
import org.apache.spark.internal.Logging

‎resource-managers/kubernetes/core/volcano/src/test/scala/org/apache/spark/deploy/k8s/features/VolcanoFeatureStepSuite.scala‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ package org.apache.spark.deploy.k8s.features
1818

1919
import java.io.File
2020

21-
import io.fabric8.volcano.scheduling.v1beta1.PodGroup
21+
import io.fabric8.volcano.api.model.scheduling.v1beta1.PodGroup
2222

2323
import org.apache.spark.{SparkConf, SparkFunSuite}
2424
import org.apache.spark.deploy.k8s._

‎resource-managers/kubernetes/integration-tests/volcano/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/VolcanoTestsSuite.scala‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,8 +28,8 @@ import scala.concurrent.Future
2828
import scala.jdk.CollectionConverters._
2929

3030
import io.fabric8.kubernetes.api.model.{HasMetadata, Pod, Quantity}
31+
import io.fabric8.volcano.api.model.scheduling.v1beta1.{Queue, QueueBuilder}
3132
import io.fabric8.volcano.client.VolcanoClient
32-
import io.fabric8.volcano.scheduling.v1beta1.{Queue, QueueBuilder}
3333
import org.scalatest.BeforeAndAfterEach
3434
import org.scalatest.concurrent.Eventually
3535

0 commit comments

Comments
 (0)