Skip to content

HADOOP-13551. AWS metrics wire-up (#2778) #2815

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

HADOOP-13551. AWS metrics wire-up (#2778) #2815

wants to merge 2 commits into from

Conversation

steveloughran
Copy link
Contributor

Backport of trunk #2778 to 3.3; had to include another patch to avoid merge conflict.

Not yet done any s3 tests

mukund-thakur and others added 2 commits March 24, 2021 16:53
@mukund-thakur @steveloughran
HADOOP-17305. Fix ITestCustomSigner to work with s3 compatible endpoi…
193536f 
…nts (apache#2395)


Contributed by Mukund Thakur
@steveloughran
HADOOP-13551. AWS metrics wire-up (apache#2778)
bb15dae 
Moves to the builder API for AWS S3 client creation, and
offers a similar style of API to the S3A FileSystem and tests, hiding
the details of which options are client, which are in AWS Conf,
and doing the wiring up of S3A statistics interfaces to the AWS
SDK internals. S3A Statistics, including IOStatistics, should now
count throttling events handled in the AWS SDK itself.

This patch restores endpoint determination by probes to US-East-1
if the client isn't configured with fs.s3a.endpoint.

Explicitly setting the endpoint will save the cost of these probe
HTTP requests.

Contributed by Steve Loughran.
@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 21m 29s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 5 new or modified test files.
_ branch-3.3 Compile Tests _
+1 💚 mvninstall 32m 30s branch-3.3 passed
+1 💚 compile 0m 38s branch-3.3 passed
+1 💚 checkstyle 0m 31s branch-3.3 passed
+1 💚 mvnsite 0m 46s branch-3.3 passed
+1 💚 javadoc 0m 34s branch-3.3 passed
+1 💚 spotbugs 1m 9s branch-3.3 passed
+1 💚 shadedclient 16m 19s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 38s the patch passed
+1 💚 compile 0m 32s the patch passed
+1 💚 javac 0m 32s the patch passed
+1 💚 blanks 0m 1s The patch has no blanks issues.
+1 💚 checkstyle 0m 21s hadoop-tools/hadoop-aws: The patch generated 0 new + 17 unchanged - 1 fixed = 17 total (was 18)
+1 💚 mvnsite 0m 33s the patch passed
+1 💚 javadoc 0m 24s the patch passed
+1 💚 spotbugs 1m 9s the patch passed
+1 💚 shadedclient 15m 47s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 1m 59s hadoop-aws in the patch passed.
+1 💚 asflicense 0m 35s The patch does not generate ASF License warnings.
96m 35s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2815/1/artifact/out/Dockerfile
GITHUB PR #2815
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux a7216a827e9d 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / bb15dae
Default Java Private Build-1.8.0_282-8u282-b08-0ubuntu1~18.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2815/1/testReport/
Max. process+thread count 534 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2815/1/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@steveloughran
Copy link
Contributor Author

tested, merged in and pushed up

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@steveloughran @hadoop-yetus @mukund-thakur