Skip to content

Commit afda510

Browse files
prabhjyotsinghpanbingkundongjoon-hyunbjornjorgensenroczei
authored
ODP-3237: Fix CVE-2023-3635 com.squareup.okio:okio / kubernetes-client in spark (#66)
* ODP-3237: Fix compile * [SPARK-44537][BUILD] Upgrade kubernetes-client to 6.8.0 The pr aims to upgrade kubernetes-client from 6.7.2 to 6.8.0. - The newest version brings some bug fixed & improvment, eg: Fix fabric8io/kubernetes-client#5221: Empty kube config file causes NPE Fix fabric8io/kubernetes-client#5281: Ensure the KubernetesCrudDispatcher's backing map is accessed w/lock Fix fabric8io/kubernetes-client#5298: Prevent requests needing authentication from causing a 403 response Fix fabric8io/kubernetes-client#5233: Generalized SchemaSwap to allow for cycle expansion Fix fabric8io/kubernetes-client#5262: all built-in collections will omit empty in their serialized form. - The full release notes: https://github.com/fabric8io/kubernetes-client/releases/ No. Pass GA. Closes apache#42142 from panbingkun/SPARK-44537. Authored-by: panbingkun <pbk1982@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 6b6216c) * [SPARK-44821][BUILD][K8S] Upgrade `kubernetes-client` to 6.8.1 ### What changes were proposed in this pull request? This PR aims to upgrade kubernetes-client to 6.8.1 ### Why are the changes needed? To bring two additional bug fixes. - https://github.com/fabric8io/kubernetes-client/releases/tag/v6.8.1 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. Closes apache#42505 from dongjoon-hyun/SPARK-44821. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Kent Yao <yao@apache.org> (cherry picked from commit e25a654) * [SPARK-45465][BUILD][K8S] Upgrade `kubernetes-client` to 6.9.0 for K8s 1.28 This PR aims to upgrade `kubernetes-client` to 6.9.0 for K8s 1.28. `kubernetes-client` 6.9.0 includes new features and bug fixes on top of K8s 1.23 model. - https://github.com/fabric8io/kubernetes-client/releases/tag/v6.9.0 - fabric8io/kubernetes-client#5401 No. Pass the CIs. No. Closes apache#43284 from dongjoon-hyun/SPARK-45465. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 89acebb) * [SPARK-45728][BUILD][K8S] Upgrade `kubernetes-client` to 6.9.1 ### What changes were proposed in this pull request? This PR aims to upgrade `kubernetes-client` to 6.9.1. ### Why are the changes needed? This will bring us the following bug fixes including the CRD generator fix. - https://github.com/fabric8io/kubernetes-client/releases/tag/v6.9.1 - Fix fabric8io/kubernetes-client#5501: (crd-generator) Fix fallback value of Default annotation in presence of multiple accessors ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#43589 from dongjoon-hyun/SPARK-45728. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 311602a) * [SPARK-46662][K8S][BUILD] Upgrade `kubernetes-client` to 6.10.0 Upgrade `kubernetes-client` from 6.9.1 to 6.10.0 [Release notes 6.10.0](https://github.com/fabric8io/kubernetes-client/releases/tag/v6.10.0) [Release notes 6.9.2](https://github.com/fabric8io/kubernetes-client/releases/tag/v6.9.2) [Updated okio to version 1.17.6 to avoid CVE-2023-3635](fabric8io/kubernetes-client#5587) [Upgrade Kubernetes Model to Kubernetes v1.29.0](fabric8io/kubernetes-client#5686) No. Pass GA No. Closes apache#44672 from bjornjorgensen/kubclient6.10. Authored-by: Bjørn Jørgensen <bjornjorgensen@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit dcfd37c) * [SPARK-47550][K8S][BUILD] Update `kubernetes-client` to 6.11.0 ### What changes were proposed in this pull request? Update `kubernetes-client` from 6.10.0 to 6.11.0 ### Why are the changes needed? [Release notes for 6.11.0](https://github.com/fabric8io/kubernetes-client/releases/tag/v6.11.0) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#45707 from bjornjorgensen/kub-client6.11.0. Authored-by: Bjørn Jørgensen <bjornjorgensen@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 7b9b3cb) * [SPARK-47860][BUILD][K8S] Upgrade `kubernetes-client` to 6.12.0 ### What changes were proposed in this pull request? This PR aims to upgrade `kubernetes-client` to 6.12.0. ### Why are the changes needed? To bring the latest bug fixes. - https://github.com/fabric8io/kubernetes-client/releases/tag/v6.12.0 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#46066 from dongjoon-hyun/SPARK-47860. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit becbca6) * [SPARK-47915][BUILD][K8S] Upgrade `kubernetes-client` to 6.12.1 ### What changes were proposed in this pull request? Upgrade `kubernetes-client` from 6.12.0 to 6.12.1 ### Why are the changes needed? [Release notes](https://github.com/fabric8io/kubernetes-client/releases/tag/v6.12.1) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#46137 from bjornjorgensen/kub-client6.12.1. Authored-by: Bjørn Jørgensen <bjornjorgensen@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 8aa8ad6) * [SPARK-48514][BUILD][K8S] Upgrade `kubernetes-client` to 6.13.0 ### What changes were proposed in this pull request? Upgrade kubernetes-client from 6.12.1 to 6.13.0 ### Why are the changes needed? Upgrade Fabric8 Kubernetes Model to Kubernetes v1.30.0 [Release log 6.13.0](https://github.com/fabric8io/kubernetes-client/releases/tag/v6.13.0) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#46854 from bjornjorgensen/kubclient6.13.0. Authored-by: Bjørn Jørgensen <bjornjorgensen@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> (cherry picked from commit 6475ddf) * [SPARK-48801][BUILD][K8S] Upgrade `kubernetes-client` to 6.13.1 ### What changes were proposed in this pull request? The pr aims to upgrade `kubernetes-client` from `6.13.0` to `6.13.1`. ### Why are the changes needed? - The full release notes: https://github.com/fabric8io/kubernetes-client/releases/tag/v6.13.1 - The newest version fixed some bug, eg: Fix fabric8io/kubernetes-client#6059: Swallow rejected execution from internal usage of the informer executor Fix fabric8io/kubernetes-client#6068: KubernetesMockServer provides incomplete Configuration while creating test Config for KubernetesClient Fix fabric8io/kubernetes-client#6085: model getters have same annotations as fields (breaks native) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#47206 from panbingkun/SPARK-48801. Authored-by: panbingkun <panbingkun@baidu.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 5b00786) * [SPARK-49196][BUILD] Upgrade `kubernetes-client` to 6.13.2 ### What changes were proposed in this pull request? The pr aims to upgrade `kubernetes-client` from `6.13.1` to `6.13.2`. ### Why are the changes needed? - The full release notes: https://github.com/fabric8io/kubernetes-client/releases/tag/v6.13.2 - The newest version fixed some bug, eg: Fix fabric8io/kubernetes-client#6066: Added support for missing v1.APIVersions in KubernetesClient Fix fabric8io/kubernetes-client#6110: VolumeSource (and other file mode fields) in Octal are correctly interpreted Fix fabric8io/kubernetes-client#6137: ConfigBuilder.withAutoConfigure is not working ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#47703 from panbingkun/SPARK-49196. Authored-by: panbingkun <panbingkun@baidu.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit d09e1c5) * [SPARK-45590][BUILD] Upgrade okio to 1.17.6 from 1.15.0 This PR aims to upgrade `okio` from 1.15.0 to 1.17.6. Okio 1.15.0 is vulnerable due to CVE-2023-3635, details: https://nvd.nist.gov/vuln/detail/CVE-2023-3635 Previous attempts to fix this security issue: Update okio to version 1.17.6 apache#5587: fabric8io/kubernetes-client#5587 Followup to Update okio to version 1.17.6 apache#5935: fabric8io/kubernetes-client#5935 Unfortunately it is still using 1.15.0: https://github.com/apache/spark/blob/v4.0.0-preview1/dev/deps/spark-deps-hadoop-3-hive-2.3#L227 https://github.com/apache/spark/blob/v3.5.2/dev/deps/spark-deps-hadoop-3-hive-2.3#L210 No. Pass the CIs. No. Closes apache#47758 from roczei/SPARK-45590. Authored-by: Gabor Roczei <roczei@cloudera.com> Signed-off-by: Kent Yao <yao@apache.org> (cherry picked from commit c8cf394) * [SPARK-49255][BUILD][K8S] Upgrade `kubernetes-client` to 6.13.3 ### What changes were proposed in this pull request? This PR aims to upgrade `kubernetes-client` to 6.13.3. ### Why are the changes needed? Unfortunately, there is an important bug fix after we upgraded to 6.13.2. This PR aims to bring the fix. - https://github.com/fabric8io/kubernetes-client/releases/tag/v6.13.3 - fabric8io/kubernetes-client#6249 - fabric8io/kubernetes-client@1a66228 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#47782 from dongjoon-hyun/SPARK-49255. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 7a670d7) * [SPARK-49800][BUILD][K8S] Upgrade `kubernetes-client` to 6.13.4 ### What changes were proposed in this pull request? Upgrade `kubernetes-client` from 6.13.3 to 6.13.4 ### Why are the changes needed? New version that have 5 fixes [Release log 6.13.4](https://github.com/fabric8io/kubernetes-client/releases/tag/v6.13.4) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#48268 from bjornjorgensen/k8sclient6.13.4. Authored-by: Bjørn Jørgensen <bjornjorgensen@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 54e62a1) * [SPARK-50493][SPARK-37687][BUILD] Migrate kubernetes-client from `6.x` to `7.x` The pr aims to migrate kubernetes-client from `6.x` to `7.x`, includes: - upgrade `kubernetes-client` from `6.13.2` to `7.0.1`. - switch default `HttpClient` implementation from `OkHttp` to `Vert.x`. - The full release notes: https://github.com/fabric8io/kubernetes-client/releases/tag/v7.0.1 https://github.com/fabric8io/kubernetes-client/releases/tag/v7.0.0 - [Vert.x as default HttpClient implementation](https://github.com/fabric8io/kubernetes-client/blob/v7.0.0/doc/MIGRATION-v7.md#vertx-as-default-httpclient-implementation-) No. Pass GA. No. Closes apache#49159 from panbingkun/k8s_client_7_vert_x. Authored-by: panbingkun <panbingkun@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit dccb129) --------- Co-authored-by: panbingkun <pbk1982@gmail.com> Co-authored-by: Dongjoon Hyun <dongjoon@apache.org> Co-authored-by: Dongjoon Hyun <dhyun@apple.com> Co-authored-by: Bjørn Jørgensen <bjornjorgensen@gmail.com> Co-authored-by: panbingkun <panbingkun@baidu.com> Co-authored-by: Gabor Roczei <roczei@cloudera.com> Co-authored-by: panbingkun <panbingkun@apache.org>
1 parent 8c39433 commit afda510

File tree

16 files changed

+78
-60
lines changed

16 files changed

+78
-60
lines changed

‎LICENSE-binary‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -214,6 +214,10 @@ com.googlecode.javaewah:JavaEWAH
214214
com.twitter:chill-java
215215
com.twitter:chill_2.12
216216
com.univocity:univocity-parsers
217+
io.vertx:vertx-auth-common
218+
io.vertx:vertx-core
219+
io.vertx:vertx-web-client
220+
io.vertx:vertx-web-common
217221
javax.jdo:jdo-api
218222
joda-time:joda-time
219223
net.sf.opencsv:opencsv

‎connector/docker-integration-tests/pom.xml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,12 @@
111111
<groupId>org.apache.hadoop</groupId>
112112
<artifactId>hadoop-minikdc</artifactId>
113113
<scope>test</scope>
114+
<exclusions>
115+
<exclusion>
116+
<groupId>org.jline</groupId>
117+
<artifactId>jline</artifactId>
118+
</exclusion>
119+
</exclusions>
114120
</dependency>
115121
<!-- Although SPARK-28737 upgraded Jersey to 2.29 for JDK11, 'com.spotify.docker-client' still
116122
uses this repackaged 'jersey-guava'. We add this back for JDK8/JDK11 testing. -->

‎connector/kafka-0-10-sql/pom.xml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,12 @@
123123
<dependency>
124124
<groupId>org.apache.hadoop</groupId>
125125
<artifactId>hadoop-minikdc</artifactId>
126+
<exclusions>
127+
<exclusion>
128+
<groupId>org.jline</groupId>
129+
<artifactId>jline</artifactId>
130+
</exclusion>
131+
</exclusions>
126132
</dependency>
127133
<!-- Kafka embedded server uses Zookeeper 3.5.7 API -->
128134
<dependency>

‎core/pom.xml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -450,6 +450,12 @@
450450
<groupId>org.apache.hadoop</groupId>
451451
<artifactId>hadoop-minikdc</artifactId>
452452
<scope>test</scope>
453+
<exclusions>
454+
<exclusion>
455+
<groupId>org.jline</groupId>
456+
<artifactId>jline</artifactId>
457+
</exclusion>
458+
</exclusions>
453459
</dependency>
454460
<dependency>
455461
<groupId>net.razorvine</groupId>

‎dev/deps/spark-deps-hadoop-3-hive-2.3‎

Lines changed: 34 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -144,31 +144,31 @@ jsr305/3.0.0//jsr305-3.0.0.jar
144144
jta/1.1//jta-1.1.jar
145145
jul-to-slf4j/2.0.7//jul-to-slf4j-2.0.7.jar
146146
kryo-shaded/4.0.2//kryo-shaded-4.0.2.jar
147-
kubernetes-client-api/6.8.1//kubernetes-client-api-6.8.1.jar
148-
kubernetes-client/6.8.1//kubernetes-client-6.8.1.jar
149-
kubernetes-httpclient-okhttp/6.8.1//kubernetes-httpclient-okhttp-6.8.1.jar
150-
kubernetes-model-admissionregistration/6.8.1//kubernetes-model-admissionregistration-6.8.1.jar
151-
kubernetes-model-apiextensions/6.8.1//kubernetes-model-apiextensions-6.8.1.jar
152-
kubernetes-model-apps/6.8.1//kubernetes-model-apps-6.8.1.jar
153-
kubernetes-model-autoscaling/6.8.1//kubernetes-model-autoscaling-6.8.1.jar
154-
kubernetes-model-batch/6.8.1//kubernetes-model-batch-6.8.1.jar
155-
kubernetes-model-certificates/6.8.1//kubernetes-model-certificates-6.8.1.jar
156-
kubernetes-model-common/6.8.1//kubernetes-model-common-6.8.1.jar
157-
kubernetes-model-coordination/6.8.1//kubernetes-model-coordination-6.8.1.jar
158-
kubernetes-model-core/6.8.1//kubernetes-model-core-6.8.1.jar
159-
kubernetes-model-discovery/6.8.1//kubernetes-model-discovery-6.8.1.jar
160-
kubernetes-model-events/6.8.1//kubernetes-model-events-6.8.1.jar
161-
kubernetes-model-extensions/6.8.1//kubernetes-model-extensions-6.8.1.jar
162-
kubernetes-model-flowcontrol/6.8.1//kubernetes-model-flowcontrol-6.8.1.jar
163-
kubernetes-model-gatewayapi/6.8.1//kubernetes-model-gatewayapi-6.8.1.jar
164-
kubernetes-model-metrics/6.8.1//kubernetes-model-metrics-6.8.1.jar
165-
kubernetes-model-networking/6.8.1//kubernetes-model-networking-6.8.1.jar
166-
kubernetes-model-node/6.8.1//kubernetes-model-node-6.8.1.jar
167-
kubernetes-model-policy/6.8.1//kubernetes-model-policy-6.8.1.jar
168-
kubernetes-model-rbac/6.8.1//kubernetes-model-rbac-6.8.1.jar
169-
kubernetes-model-resource/6.8.1//kubernetes-model-resource-6.8.1.jar
170-
kubernetes-model-scheduling/6.8.1//kubernetes-model-scheduling-6.8.1.jar
171-
kubernetes-model-storageclass/6.8.1//kubernetes-model-storageclass-6.8.1.jar
147+
kubernetes-client-api/7.0.1//kubernetes-client-api-7.0.1.jar
148+
kubernetes-client/7.0.1//kubernetes-client-7.0.1.jar
149+
kubernetes-httpclient-vertx/7.0.1//kubernetes-httpclient-vertx-7.0.1.jar
150+
kubernetes-model-admissionregistration/7.0.1//kubernetes-model-admissionregistration-7.0.1.jar
151+
kubernetes-model-apiextensions/7.0.1//kubernetes-model-apiextensions-7.0.1.jar
152+
kubernetes-model-apps/7.0.1//kubernetes-model-apps-7.0.1.jar
153+
kubernetes-model-autoscaling/7.0.1//kubernetes-model-autoscaling-7.0.1.jar
154+
kubernetes-model-batch/7.0.1//kubernetes-model-batch-7.0.1.jar
155+
kubernetes-model-certificates/7.0.1//kubernetes-model-certificates-7.0.1.jar
156+
kubernetes-model-common/7.0.1//kubernetes-model-common-7.0.1.jar
157+
kubernetes-model-coordination/7.0.1//kubernetes-model-coordination-7.0.1.jar
158+
kubernetes-model-core/7.0.1//kubernetes-model-core-7.0.1.jar
159+
kubernetes-model-discovery/7.0.1//kubernetes-model-discovery-7.0.1.jar
160+
kubernetes-model-events/7.0.1//kubernetes-model-events-7.0.1.jar
161+
kubernetes-model-extensions/7.0.1//kubernetes-model-extensions-7.0.1.jar
162+
kubernetes-model-flowcontrol/7.0.1//kubernetes-model-flowcontrol-7.0.1.jar
163+
kubernetes-model-gatewayapi/7.0.1//kubernetes-model-gatewayapi-7.0.1.jar
164+
kubernetes-model-metrics/7.0.1//kubernetes-model-metrics-7.0.1.jar
165+
kubernetes-model-networking/7.0.1//kubernetes-model-networking-7.0.1.jar
166+
kubernetes-model-node/7.0.1//kubernetes-model-node-7.0.1.jar
167+
kubernetes-model-policy/7.0.1//kubernetes-model-policy-7.0.1.jar
168+
kubernetes-model-rbac/7.0.1//kubernetes-model-rbac-7.0.1.jar
169+
kubernetes-model-resource/7.0.1//kubernetes-model-resource-7.0.1.jar
170+
kubernetes-model-scheduling/7.0.1//kubernetes-model-scheduling-7.0.1.jar
171+
kubernetes-model-storageclass/7.0.1//kubernetes-model-storageclass-7.0.1.jar
172172
lapack/3.0.3//lapack-3.0.3.jar
173173
leveldbjni-all/1.8//leveldbjni-all-1.8.jar
174174
libfb303/0.9.3//libfb303-0.9.3.jar
@@ -177,7 +177,6 @@ log4j-1.2-api/2.20.0//log4j-1.2-api-2.20.0.jar
177177
log4j-api/2.20.0//log4j-api-2.20.0.jar
178178
log4j-core/2.20.0//log4j-core-2.20.0.jar
179179
log4j-slf4j2-impl/2.20.0//log4j-slf4j2-impl-2.20.0.jar
180-
logging-interceptor/3.12.12//logging-interceptor-3.12.12.jar
181180
lz4-java/1.8.0//lz4-java-1.8.0.jar
182181
mesos/1.4.3/shaded-protobuf/mesos-1.4.3-shaded-protobuf.jar
183182
metrics-core/4.2.19//metrics-core-4.2.19.jar
@@ -188,13 +187,15 @@ metrics-jvm/4.2.19//metrics-jvm-4.2.19.jar
188187
minlog/1.3.0//minlog-1.3.0.jar
189188
netty-all/4.1.108.Final//netty-all-4.1.108.Final.jar
190189
netty-buffer/4.1.108.Final//netty-buffer-4.1.108.Final.jar
190+
netty-codec-dns/4.1.108.Final//netty-codec-dns-4.1.108.Final.jar
191191
netty-codec-http/4.1.108.Final//netty-codec-http-4.1.108.Final.jar
192192
netty-codec-http2/4.1.108.Final//netty-codec-http2-4.1.108.Final.jar
193193
netty-codec-socks/4.1.108.Final//netty-codec-socks-4.1.108.Final.jar
194194
netty-codec/4.1.108.Final//netty-codec-4.1.108.Final.jar
195195
netty-common/4.1.108.Final//netty-common-4.1.108.Final.jar
196196
netty-handler-proxy/4.1.108.Final//netty-handler-proxy-4.1.108.Final.jar
197197
netty-handler/4.1.108.Final//netty-handler-4.1.108.Final.jar
198+
netty-resolver-dns/4.1.108.Final//netty-resolver-dns-4.1.108.Final.jar
198199
netty-resolver/4.1.108.Final//netty-resolver-4.1.108.Final.jar
199200
netty-transport-classes-epoll/4.1.108.Final//netty-transport-classes-epoll-4.1.108.Final.jar
200201
netty-transport-classes-kqueue/4.1.108.Final//netty-transport-classes-kqueue-4.1.108.Final.jar
@@ -206,7 +207,7 @@ netty-transport-native-unix-common/4.1.108.Final//netty-transport-native-unix-co
206207
netty-transport/4.1.108.Final//netty-transport-4.1.108.Final.jar
207208
objenesis/3.3//objenesis-3.3.jar
208209
okhttp/3.12.12//okhttp-3.12.12.jar
209-
okio/1.15.0//okio-1.15.0.jar
210+
okio/1.17.6//okio-1.17.6.jar
210211
opencsv/2.3//opencsv-2.3.jar
211212
opentracing-api/0.33.0//opentracing-api-0.33.0.jar
212213
opentracing-noop/0.33.0//opentracing-noop-0.33.0.jar
@@ -235,7 +236,7 @@ scala-reflect/2.12.18//scala-reflect-2.12.18.jar
235236
scala-xml_2.12/2.2.0//scala-xml_2.12-2.2.0.jar
236237
shims/0.9.45//shims-0.9.45.jar
237238
slf4j-api/2.0.7//slf4j-api-2.0.7.jar
238-
snakeyaml-engine/2.6//snakeyaml-engine-2.6.jar
239+
snakeyaml-engine/2.8//snakeyaml-engine-2.8.jar
239240
snakeyaml/2.0//snakeyaml-2.0.jar
240241
snappy-java/1.1.10.3//snappy-java-1.1.10.3.jar
241242
spire-macros_2.12/0.17.0//spire-macros_2.12-0.17.0.jar
@@ -250,11 +251,15 @@ tink/1.9.0//tink-1.9.0.jar
250251
transaction-api/1.1//transaction-api-1.1.jar
251252
txw2/3.0.2//txw2-3.0.2.jar
252253
univocity-parsers/2.9.1//univocity-parsers-2.9.1.jar
254+
vertx-auth-common/4.5.11//vertx-auth-common-4.5.11.jar
255+
vertx-core/4.5.11//vertx-core-4.5.11.jar
256+
vertx-web-client/4.5.11//vertx-web-client-4.5.11.jar
257+
vertx-web-common/4.5.11//vertx-web-common-4.5.11.jar
253258
wildfly-openssl/1.1.3.Final//wildfly-openssl-1.1.3.Final.jar
254259
xbean-asm9-shaded/4.23//xbean-asm9-shaded-4.23.jar
255260
xmlschema-core/2.3.0//xmlschema-core-2.3.0.jar
256261
xz/1.9//xz-1.9.jar
257-
zjsonpatch/0.3.0//zjsonpatch-0.3.0.jar
262+
zjsonpatch/7.0.1//zjsonpatch-7.0.1.jar
258263
zookeeper-jute/3.6.3//zookeeper-jute-3.6.3.jar
259264
zookeeper/3.6.3//zookeeper-3.6.3.jar
260265
zstd-jni/1.5.5-5//zstd-jni-1.5.5-5.jar

‎hadoop-cloud/pom.xml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,8 @@
3434
</description>
3535
<properties>
3636
<sbt.project.name>hadoop-cloud</sbt.project.name>
37+
<okhttp.version>3.12.12</okhttp.version>
38+
<okio.version>1.17.6</okio.version>
3739
</properties>
3840

3941
<build>
@@ -205,6 +207,16 @@
205207
</exclusion>
206208
</exclusions>
207209
</dependency>
210+
<dependency>
211+
<groupId>com.squareup.okhttp3</groupId>
212+
<artifactId>okhttp</artifactId>
213+
<version>${okhttp.version}</version>
214+
</dependency>
215+
<dependency>
216+
<groupId>com.squareup.okio</groupId>
217+
<artifactId>okio</artifactId>
218+
<version>${okio.version}</version>
219+
</dependency>
208220
</dependencies>
209221

210222
<profiles>

‎pom.xml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -238,7 +238,7 @@
238238

239239
<!-- org.fusesource.leveldbjni will be used except on arm64 platform. -->
240240
<leveldbjni.group>org.fusesource.leveldbjni</leveldbjni.group>
241-
<kubernetes-client.version>6.7.2</kubernetes-client.version>
241+
<kubernetes-client.version>7.0.1</kubernetes-client.version>
242242

243243
<test.java.home>${java.home}</test.java.home>
244244

‎resource-managers/kubernetes/core/pom.xml‎

Lines changed: 1 addition & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@
4141
<dependencies>
4242
<dependency>
4343
<groupId>io.fabric8</groupId>
44-
<artifactId>volcano-model-v1beta1</artifactId>
44+
<artifactId>volcano-model</artifactId>
4545
<version>${kubernetes-client.version}</version>
4646
</dependency>
4747
<dependency>
@@ -75,11 +75,6 @@
7575
<scope>test</scope>
7676
</dependency>
7777

78-
<dependency>
79-
<groupId>io.fabric8</groupId>
80-
<artifactId>kubernetes-httpclient-okhttp</artifactId>
81-
<version>${kubernetes-client.version}</version>
82-
</dependency>
8378
<dependency>
8479
<groupId>io.fabric8</groupId>
8580
<artifactId>kubernetes-client</artifactId>

‎resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala‎

Lines changed: 1 addition & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -24,16 +24,12 @@ import com.google.common.io.Files
2424
import io.fabric8.kubernetes.client.{ConfigBuilder, KubernetesClient, KubernetesClientBuilder}
2525
import io.fabric8.kubernetes.client.Config.KUBERNETES_REQUEST_RETRY_BACKOFFLIMIT_SYSTEM_PROPERTY
2626
import io.fabric8.kubernetes.client.Config.autoConfigure
27-
import io.fabric8.kubernetes.client.okhttp.OkHttpClientFactory
2827
import io.fabric8.kubernetes.client.utils.Utils.getSystemPropertyOrEnvVar
29-
import okhttp3.Dispatcher
30-
import okhttp3.OkHttpClient
3128

3229
import org.apache.spark.SparkConf
3330
import org.apache.spark.deploy.k8s.Config._
3431
import org.apache.spark.internal.Logging
3532
import org.apache.spark.internal.config.ConfigEntry
36-
import org.apache.spark.util.ThreadUtils
3733

3834
/**
3935
* Spark-opinionated builder for Kubernetes clients. It uses a prefix plus common suffixes to
@@ -67,10 +63,6 @@ private[spark] object SparkKubernetesClientFactory extends Logging {
6763
.getOption(s"$kubernetesAuthConfPrefix.$CLIENT_KEY_FILE_CONF_SUFFIX")
6864
val clientCertFile = sparkConf
6965
.getOption(s"$kubernetesAuthConfPrefix.$CLIENT_CERT_FILE_CONF_SUFFIX")
70-
// TODO(SPARK-37687): clean up direct usage of OkHttpClient, see also:
71-
// https://github.com/fabric8io/kubernetes-client/issues/3547
72-
val dispatcher = new Dispatcher(
73-
ThreadUtils.newDaemonCachedThreadPool("kubernetes-dispatcher"))
7466

7567
// Allow for specifying a context used to auto-configure from the users K8S config file
7668
val kubeContext = sparkConf.get(KUBERNETES_CONTEXT).filter(_.nonEmpty)
@@ -106,17 +98,9 @@ private[spark] object SparkKubernetesClientFactory extends Logging {
10698
}.withOption(namespace) {
10799
(ns, configBuilder) => configBuilder.withNamespace(ns)
108100
}.build()
109-
val factoryWithCustomDispatcher = new OkHttpClientFactory() {
110-
override protected def additionalConfig(builder: OkHttpClient.Builder): Unit = {
111-
builder.dispatcher(dispatcher)
112-
}
113-
}
114101
logDebug("Kubernetes client config: " +
115102
new ObjectMapper().writerWithDefaultPrettyPrinter().writeValueAsString(config))
116-
new KubernetesClientBuilder()
117-
.withHttpClientFactory(factoryWithCustomDispatcher)
118-
.withConfig(config)
119-
.build()
103+
new KubernetesClientBuilder().withConfig(config).build()
120104
}
121105

122106
private implicit class OptionConfigurableConfigBuilder(val configBuilder: ConfigBuilder)

‎resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/MountVolumesFeatureStep.scala‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,7 @@ private[spark] class MountVolumesFeatureStep(conf: KubernetesConf)
9696
.withNewSpec()
9797
.withStorageClassName(storageClass.get)
9898
.withAccessModes(accessMode)
99-
.withResources(new ResourceRequirementsBuilder()
99+
.withResources(new VolumeResourceRequirementsBuilder()
100100
.withRequests(Map("storage" -> new Quantity(size.get)).asJava).build())
101101
.endSpec()
102102
.build())

0 commit comments

Comments
 (0)