Skip to content

HADOOP-18742. AWS v2 SDK: stabilise dependencies with rest of hadoop libraries #5739

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 13 commits into from
Closed

HADOOP-18742. AWS v2 SDK: stabilise dependencies with rest of hadoop libraries #5739

wants to merge 13 commits into from

Conversation

steveloughran
Copy link
Contributor

  • explicit jackson cbor declaration in hadoop-aws to ensure hadoop version gets picked up
  • cut jackson, eventstream and ion from aws SDK dependencies

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

ahmarsuhail and others added 13 commits May 17, 2023 10:17
@ahmarsuhail @passaro
HADOOP-18073. Upgrade AWS SDK to v2 in S3A [work in progress]
f2bf08c 
See aws_sdk_v2_changelog.md for details.

Co-authored-by: Ahmar Suhail <ahmarsu@amazon.co.uk>
Co-authored-by: Alessandro Passaro <alexpax@amazon.co.uk>
@ahmarsuhail
HADOOP-18073. Address review comments. (apache#31)
68510c8 
addresses review comments + yetus errors

Co-authored-by: Ahmar Suhail <ahmarsu@amazon.co.uk>
@passaro @ahmarsuhail
Move MultiObjectDeleteException to impl
aa1cf19
@passaro @ahmarsuhail
Reinstate old constants
1ab7e68
@passaro @ahmarsuhail
Move TransferManager initialization to ClientFactory
0c2fd28
@passaro @ahmarsuhail
Add unit tests for BlockingEnumeration
38dff86
@passaro @ahmarsuhail
Add unit tests for SelectEventStreamPublisher
0338fd8
@ahmarsuhail
updates new providers in TestS3AAWSCredentialsProvider to V2
644a32f
@ahmarsuhail
update GET range referrer header logic to V2
ce02671
@ahmarsuhail
adds in unit check for bytes
fb6d12a
@ahmarsuhail
HADOOP-18565. Complete outstanding items for the AWS SDK V2 upgrade. (a…
1f137d3 
…pache#5421)

Changes include
* use bundled transfer manager
* adds transfer listener to upload
* adds support for custom signers
* don't set default endpoint
* removes v1 sdk bundle, only use core package
* implements region caching
+ many more

Note: spotbugs is warning about inconsistent
synchronization in accessing a new s3a FS field.
This will be fixed in a follow-up patch.

Contributed by Ahmar Suhail
@ahmarsuhail
fixes issues after rebase
75220b7
@steveloughran
HADOOP-18742. AWS v2 SDK: stabilise dependencies with rest of hadoop …
7978e70 
…libraries

* explicit jackson declaration in hadoop-aws
* cut jackson, eventstream and ion from aws SDK dependencies

Change-Id: I47f6e10d42c8067df8255eca69799469d7252480
@steveloughran
Copy link
Contributor Author

not tested this yet; this is a cherrypick of my existing branch onto the rebased sdk branch

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 54s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 xmllint 0m 0s xmllint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ feature-HADOOP-18073-s3a-sdk-upgrade Compile Tests _
+0 🆗 mvndep 20m 35s Maven dependency ordering for branch
+1 💚 mvninstall 24m 0s feature-HADOOP-18073-s3a-sdk-upgrade passed
+1 💚 compile 18m 36s feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚 compile 17m 2s feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private Build-1.8.0_362-8u372-gaus1-0ubuntu120.04-b09
+1 💚 mvnsite 1m 29s feature-HADOOP-18073-s3a-sdk-upgrade passed
+1 💚 javadoc 1m 11s feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚 javadoc 1m 15s feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private Build-1.8.0_362-8u372-gaus1-0ubuntu120.04-b09
+1 💚 shadedclient 108m 17s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 30s Maven dependency ordering for patch
+1 💚 mvninstall 0m 46s the patch passed
+1 💚 compile 17m 53s the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚 javac 17m 53s the patch passed
+1 💚 compile 16m 57s the patch passed with JDK Private Build-1.8.0_362-8u372-gaus1-0ubuntu120.04-b09
+1 💚 javac 16m 57s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 1m 24s the patch passed
+1 💚 javadoc 1m 5s the patch passed with JDK Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
+1 💚 javadoc 1m 15s the patch passed with JDK Private Build-1.8.0_362-8u372-gaus1-0ubuntu120.04-b09
+1 💚 shadedclient 30m 20s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 0m 30s hadoop-project in the patch passed.
+1 💚 unit 2m 50s hadoop-aws in the patch passed.
+1 💚 asflicense 1m 0s The patch does not generate ASF License warnings.
183m 49s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5739/1/artifact/out/Dockerfile
GITHUB PR #5739
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint
uname Linux 4e404fda821f 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision feature-HADOOP-18073-s3a-sdk-upgrade / 7978e70
Default Java Private Build-1.8.0_362-8u372-gaus1-0ubuntu120.04-b09
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u372-gaus1-0ubuntu120.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5739/1/testReport/
Max. process+thread count 540 (vs. ulimit of 5500)
modules C: hadoop-project hadoop-tools/hadoop-aws U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5739/1/console
versions git=2.25.1 maven=3.6.3
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

ayushtkn
ayushtkn reviewed Jun 29, 2023
View reviewed changes
Copy link
Member

@ayushtkn ayushtkn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not tested this yet

if you test it and things work then this LGTM

@asfgit asfgit force-pushed the feature-HADOOP-18073-s3a-sdk-upgrade branch from 75220b7 to acb31b6 Compare July 20, 2023 18:49
@steveloughran
Copy link
Contributor Author

pulling in to #5872 which downgrades the core lib to provided; we gain consistency by excluding unshaded stuff

@steveloughran steveloughran mentioned this pull request Jul 26, 2023
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ayushtkn ayushtkn ayushtkn left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@steveloughran @hadoop-yetus @ayushtkn @ahmarsuhail @passaro