Skip to content

Commit 492a886

Browse files
MartijnVisserMartijnVisser
authored
[FLINK-33704][Filesytems] Update GCS filesystems to latest available versions. This closes #23837
* [FLINK-33704][Filesystem] Update GCS filesystem dependencies - Update GS SDK from 2.15.0 to 2.29.1 - Update GS Hadoop Connector from 2.2.15 to 2.2.18 - Update GS GRPC to highest shared version * [FLINK-33704] Make NOTICE inclusion of com.google.re2j for GCS consistent RE2J is listed for the other filesystems as Go license, we should remain consistent throughout the project
1 parent 645afd8 commit 492a886

File tree

4 files changed

+64
-59
lines changed
  • docs
    • content/docs/deployment/filesystems
    • content.zh/docs/deployment/filesystems

4 files changed

+64
-59
lines changed

‎docs/content.zh/docs/deployment/filesystems/gcs.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ Note that these examples are *not* exhaustive and you can use GCS in other place
5555
Flink provides the `flink-gs-fs-hadoop` file system to write to GCS.
5656
This implementation is self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use it.
5757

58-
`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the *gs://* scheme. It uses Google's [gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/hadoop3-2.2.15) Hadoop library to access GCS. It also uses Google's [google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.15.0) library to provide `RecoverableWriter` support.
58+
`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the *gs://* scheme. It uses Google's [gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/hadoop3-2.2.18) Hadoop library to access GCS. It also uses Google's [google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.29.1) library to provide `RecoverableWriter` support.
5959

6060
This file system can be used with the [FileSystem connector]({{< ref "docs/connectors/datastream/filesystem.md" >}}).
6161

@@ -68,7 +68,7 @@ cp ./opt/flink-gs-fs-hadoop-{{< version >}}.jar ./plugins/gs-fs-hadoop/
6868

6969
### Configuration
7070

71-
The underlying Hadoop file system can be configured using the [Hadoop configuration keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.15/gcs/CONFIGURATION.md) for `gcs-connector` by adding the configurations to your `flink-conf.yaml`.
71+
The underlying Hadoop file system can be configured using the [Hadoop configuration keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.18/gcs/CONFIGURATION.md) for `gcs-connector` by adding the configurations to your `flink-conf.yaml`.
7272

7373
For example, `gcs-connector` has a `fs.gs.http.connect-timeout` configuration key. If you want to change it, you need to set `gs.http.connect-timeout: xyz` in `flink-conf.yaml`. Flink will internally translate this back to `fs.gs.http.connect-timeout`.
7474

‎docs/content/docs/deployment/filesystems/gcs.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ Note that these examples are *not* exhaustive and you can use GCS in other place
5555
Flink provides the `flink-gs-fs-hadoop` file system to write to GCS.
5656
This implementation is self-contained with no dependency footprint, so there is no need to add Hadoop to the classpath to use it.
5757

58-
`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the *gs://* scheme. It uses Google's [gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/hadoop3-2.2.15) Hadoop library to access GCS. It also uses Google's [google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.15.0) library to provide `RecoverableWriter` support.
58+
`flink-gs-fs-hadoop` registers a `FileSystem` wrapper for URIs with the *gs://* scheme. It uses Google's [gcs-connector](https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector/hadoop3-2.2.18) Hadoop library to access GCS. It also uses Google's [google-cloud-storage](https://mvnrepository.com/artifact/com.google.cloud/google-cloud-storage/2.29.1) library to provide `RecoverableWriter` support.
5959

6060
This file system can be used with the [FileSystem connector]({{< ref "docs/connectors/datastream/filesystem.md" >}}).
6161

@@ -68,7 +68,7 @@ cp ./opt/flink-gs-fs-hadoop-{{< version >}}.jar ./plugins/gs-fs-hadoop/
6868

6969
### Configuration
7070

71-
The underlying Hadoop file system can be configured using the [Hadoop configuration keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.15/gcs/CONFIGURATION.md) for `gcs-connector` by adding the configurations to your `flink-conf.yaml`.
71+
The underlying Hadoop file system can be configured using the [Hadoop configuration keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.18/gcs/CONFIGURATION.md) for `gcs-connector` by adding the configurations to your `flink-conf.yaml`.
7272

7373
For example, `gcs-connector` has a `fs.gs.http.connect-timeout` configuration key. If you want to change it, you need to set `gs.http.connect-timeout: xyz` in `flink-conf.yaml`. Flink will internally translate this back to `fs.gs.http.connect-timeout`.
7474

‎flink-filesystems/flink-gs-fs-hadoop/pom.xml

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -34,10 +34,10 @@ under the License.
3434
<properties>
3535
<!-- If updating these dependency versions, please also update the corresponding links -->
3636
<!-- in the GCS file system documentation. -->
37-
<fs.gs.sdk.version>2.15.0</fs.gs.sdk.version>
38-
<fs.gs.connector.version>hadoop3-2.2.15</fs.gs.connector.version>
37+
<fs.gs.sdk.version>2.29.1</fs.gs.sdk.version>
38+
<fs.gs.connector.version>hadoop3-2.2.18</fs.gs.connector.version>
3939
<!-- Set this to the highest version of grpc artifacts from gcs-connector and google-cloud-storage -->
40-
<fs.gs.grpc.version>1.50.3</fs.gs.grpc.version>
40+
<fs.gs.grpc.version>1.59.1</fs.gs.grpc.version>
4141
</properties>
4242

4343
<dependencies>
@@ -98,6 +98,10 @@ under the License.
9898
<groupId>com.google.guava</groupId>
9999
<artifactId>failureaccess</artifactId>
100100
</exclusion>
101+
<exclusion>
102+
<groupId>com.google.errorprone</groupId>
103+
<artifactId>error_prone_annotations</artifactId>
104+
</exclusion>
101105
<!-- exclude dependency because of its GPLv2 license, see https://github.com/apache/flink/pull/15599#issuecomment-850241316 -->
102106
<exclusion>
103107
<groupId>javax.annotation</groupId>

‎flink-filesystems/flink-gs-fs-hadoop/src/main/resources/META-INF/NOTICE

Lines changed: 53 additions & 52 deletions
Original file line numberDiff line numberDiff line change
@@ -9,53 +9,55 @@ This project bundles the following dependencies under the Apache Software Licens
99
- com.fasterxml.jackson.core:jackson-core:2.15.3
1010
- com.google.android:annotations:4.1.1.4
1111
- com.google.api-client:google-api-client-jackson2:2.0.1
12-
- com.google.api-client:google-api-client:2.0.1
13-
- com.google.api.grpc:gapic-google-cloud-storage-v2:2.15.0-alpha
14-
- com.google.api.grpc:grpc-google-cloud-storage-v2:2.15.0-alpha
15-
- com.google.api.grpc:grpc-google-iam-v1:1.6.7
12+
- com.google.api-client:google-api-client:2.2.0
13+
- com.google.api.grpc:gapic-google-cloud-storage-v2:2.29.1-alpha
14+
- com.google.api.grpc:grpc-google-cloud-storage-v2:2.29.1-alpha
1615
- com.google.api.grpc:proto-google-cloud-monitoring-v3:1.64.0
17-
- com.google.api.grpc:proto-google-cloud-storage-v2:2.15.0-alpha
18-
- com.google.api.grpc:proto-google-common-protos:2.10.0
19-
- com.google.api.grpc:proto-google-iam-v1:1.6.7
16+
- com.google.api.grpc:proto-google-cloud-storage-v2:2.29.1-alpha
17+
- com.google.api.grpc:proto-google-common-protos:2.28.0
18+
- com.google.api.grpc:proto-google-iam-v1:1.23.0
2019
- com.google.apis:google-api-services-iamcredentials:v1-rev20211203-2.0.0
21-
- com.google.apis:google-api-services-storage:v1-rev20220705-2.0.0
22-
- com.google.auto.value:auto-value-annotations:1.10
23-
- com.google.cloud.bigdataoss:gcs-connector:hadoop3-2.2.15
24-
- com.google.cloud.bigdataoss:gcsio:2.2.15
25-
- com.google.cloud.bigdataoss:util-hadoop:hadoop3-2.2.15
26-
- com.google.cloud.bigdataoss:util:2.2.15
27-
- com.google.cloud:google-cloud-core-grpc:2.8.27
28-
- com.google.cloud:google-cloud-core-http:2.8.27
29-
- com.google.cloud:google-cloud-core:2.8.27
20+
- com.google.apis:google-api-services-storage:v1-rev20231028-2.0.0
21+
- com.google.auto.value:auto-value-annotations:1.10.4
22+
- com.google.cloud.bigdataoss:gcs-connector:hadoop3-2.2.18
23+
- com.google.cloud.bigdataoss:gcsio:2.2.18
24+
- com.google.cloud.bigdataoss:util-hadoop:hadoop3-2.2.18
25+
- com.google.cloud.bigdataoss:util:2.2.18
26+
- com.google.cloud:google-cloud-core-grpc:2.27.0
27+
- com.google.cloud:google-cloud-core-http:2.27.0
28+
- com.google.cloud:google-cloud-core:2.27.0
3029
- com.google.cloud:google-cloud-monitoring:1.82.0
31-
- com.google.cloud:google-cloud-storage:2.15.0
32-
- com.google.code.gson:gson:2.10
30+
- com.google.cloud:google-cloud-storage:2.29.1
31+
- com.google.code.gson:gson:2.10.1
3332
- com.google.flogger:flogger-system-backend:0.7.1
3433
- com.google.flogger:flogger:0.7.1
3534
- com.google.flogger:google-extensions:0.7.1
3635
- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
37-
- com.google.http-client:google-http-client-apache-v2:1.42.3
38-
- com.google.http-client:google-http-client-appengine:1.42.3
39-
- com.google.http-client:google-http-client-gson:1.42.3
40-
- com.google.http-client:google-http-client-jackson2:1.42.3
41-
- com.google.http-client:google-http-client:1.42.3
36+
- com.google.http-client:google-http-client-apache-v2:1.43.3
37+
- com.google.http-client:google-http-client-appengine:1.43.3
38+
- com.google.http-client:google-http-client-gson:1.43.3
39+
- com.google.http-client:google-http-client-jackson2:1.43.3
40+
- com.google.http-client:google-http-client:1.43.3
4241
- com.google.oauth-client:google-oauth-client:1.34.1
4342
- com.lmax:disruptor:3.4.2
4443
- commons-codec:commons-codec:1.15
45-
- io.grpc:grpc-alts:1.50.3
46-
- io.grpc:grpc-api:1.50.3
47-
- io.grpc:grpc-auth:1.50.3
48-
- io.grpc:grpc-census:1.50.3
49-
- io.grpc:grpc-context:1.50.3
50-
- io.grpc:grpc-core:1.50.3
51-
- io.grpc:grpc-googleapis:1.50.3
52-
- io.grpc:grpc-grpclb:1.50.3
53-
- io.grpc:grpc-netty-shaded:1.50.3
54-
- io.grpc:grpc-protobuf-lite:1.50.3
55-
- io.grpc:grpc-protobuf:1.50.3
56-
- io.grpc:grpc-services:1.50.3
57-
- io.grpc:grpc-stub:1.50.3
58-
- io.grpc:grpc-xds:1.50.3
44+
- io.grpc:grpc-alts:1.59.1
45+
- io.grpc:grpc-api:1.59.1
46+
- io.grpc:grpc-auth:1.59.1
47+
- io.grpc:grpc-census:1.59.1
48+
- io.grpc:grpc-context:1.59.1
49+
- io.grpc:grpc-core:1.59.1
50+
- io.grpc:grpc-googleapis:1.59.1
51+
- io.grpc:grpc-grpclb:1.59.1
52+
- io.grpc:grpc-inprocess:1.59.1
53+
- io.grpc:grpc-netty-shaded:1.59.1
54+
- io.grpc:grpc-protobuf-lite:1.59.1
55+
- io.grpc:grpc-protobuf:1.59.1
56+
- io.grpc:grpc-rls:1.59.1
57+
- io.grpc:grpc-services:1.59.1
58+
- io.grpc:grpc-stub:1.59.1
59+
- io.grpc:grpc-util:1.59.1
60+
- io.grpc:grpc-xds:1.59.1
5961
- io.opencensus:opencensus-api:0.31.1
6062
- io.opencensus:opencensus-contrib-exemplar-util:0.31.0
6163
- io.opencensus:opencensus-contrib-grpc-metrics:0.31.0
@@ -66,30 +68,29 @@ This project bundles the following dependencies under the Apache Software Licens
6668
- io.opencensus:opencensus-impl:0.31.0
6769
- io.opencensus:opencensus-impl-core:0.31.0
6870
- io.opencensus:opencensus-proto:0.2.0
69-
- io.perfmark:perfmark-api:0.25.0
71+
- io.perfmark:perfmark-api:0.26.0
7072
- org.apache.httpcomponents:httpclient:4.5.13
7173
- org.apache.httpcomponents:httpcore:4.4.14
7274
- org.conscrypt:conscrypt-openjdk-uber:2.5.2
7375

74-
This project bundles the following dependencies under BSD-2 License (https://opensource.org/licenses/BSD-2-Clause).
76+
This project bundles the following dependencies under BSD-3 License (https://opensource.org/licenses/BSD-3-Clause).
7577
See bundled license files for details.
7678

77-
- com.google.api:api-common:2.2.2
78-
- com.google.api:gax-grpc:2.19.5
79-
- com.google.api:gax-httpjson:0.104.5
80-
- com.google.api:gax:2.19.5
79+
- com.google.api:api-common:2.20.0
80+
- com.google.api:gax-grpc:2.37.0
81+
- com.google.api:gax-httpjson:2.37.0
82+
- com.google.api:gax:2.37.0
83+
- com.google.auth:google-auth-library-credentials:1.20.0
84+
- com.google.auth:google-auth-library-oauth2-http:1.20.0
85+
- com.google.protobuf:protobuf-java-util:3.24.4
86+
- com.google.protobuf:protobuf-java:3.24.4
87+
- org.threeten:threetenbp:1.6.8
8188

82-
This project bundles the following dependencies under BSD-3 License (https://opensource.org/licenses/BSD-3-Clause).
89+
This project bundles the following dependencies under the Go License (https://golang.org/LICENSE).
8390
See bundled license files for details.
84-
85-
- com.google.auth:google-auth-library-credentials:1.12.1
86-
- com.google.auth:google-auth-library-oauth2-http:1.12.1
87-
- com.google.protobuf:protobuf-java-util:3.21.9
88-
- com.google.protobuf:protobuf-java:3.21.9
89-
- com.google.re2j:re2j:1.6
90-
- org.threeten:threetenbp:1.6.4
91+
- com.google.re2j:re2j:1.7
9192

9293
This project bundles the following dependencies under the MIT License.
9394
See bundled license files for details.
9495

95-
- org.codehaus.mojo:animal-sniffer-annotations:1.22
96+
- org.codehaus.mojo:animal-sniffer-annotations:1.23

0 commit comments

Comments
 (0)