Skip to content

[FLINK-33704][Filesytems] Update GCS filesystems to latest available versions #23837

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Nov 30, 2023

Conversation

MartijnVisser
Copy link
Contributor

What is the purpose of the change

  • Update GCS dependencies for filesystem to the latest versions

Brief change log

  • Update POM files
  • Update NOTICE files
  • Moved one transitive dependency so that its consistent throughout the Flink codebase

Verifying this change

This change is a trivial rework / code cleanup without any test coverage.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): yes
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not applicable

- Update GS SDK from 2.15.0 to 2.29.1
- Update GS Hadoop Connector from 2.2.15 to 2.2.18
- Update GS GRPC to highest shared version
…tent

RE2J is listed for the other filesystems as Go license, we should remain consistent throughout the project
@flinkbot
Copy link
Collaborator

flinkbot commented Nov 30, 2023

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@patricklucas
Copy link
Contributor

@MartijnVisser coincidentally I opened another PR (#23836) on flink-gs-fs-hadoop 20 minutes prior to yours. Maybe it makes sense to include it alongside?

Copy link
Contributor

@snuyanzin snuyanzin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general it looks good

i guess there are 2 more places requiring to update version as well

The underlying Hadoop file system can be configured using the [Hadoop configuration keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.15/gcs/CONFIGURATION.md) for `gcs-connector` by adding the configurations to your `flink-conf.yaml`.

and

The underlying Hadoop file system can be configured using the [Hadoop configuration keys](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.15/gcs/CONFIGURATION.md) for `gcs-connector` by adding the configurations to your `flink-conf.yaml`.

@MartijnVisser MartijnVisser merged commit 492a886 into apache:master Nov 30, 2023
@MartijnVisser MartijnVisser deleted the gcs-tech-debt branch November 30, 2023 14:55
MartijnVisser added a commit to MartijnVisser/flink that referenced this pull request Dec 15, 2023
…versions. This closes apache#23837

* [FLINK-33704][Filesystem] Update GCS filesystem dependencies

- Update GS SDK from 2.15.0 to 2.29.1
- Update GS Hadoop Connector from 2.2.15 to 2.2.18
- Update GS GRPC to highest shared version

* [FLINK-33704] Make NOTICE inclusion of com.google.re2j for GCS consistent

RE2J is listed for the other filesystems as Go license, we should remain consistent throughout the project

(cherry picked from commit 492a886)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants