HADOOP-13551. AWS metrics wire-up #2778
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Tested, s3 london, with the new suite to force tests against landsat and common crawl. If you comment out the That is: central endpoint's 301 responses don't trigger a switch to the endpoint. Maybe we should include that in troubleshooting |
already in it |
|
Full test run complete against s3 london; sole failure is #2777 |
|
tests against S3 ireland with endpoint option unset; bucket unversioned. Test failures are (1) transient and (2) fixed in another PR respectively. |
steveloughran
force-pushed
the
s3/HADOOP-13551-aws-metrics
branch
2 times, most recently
from
5d2075b to
0923b23
Compare
Moves to the builder API for AWS S3 client creation, and offers a similar-style API to the S3AFS and tests, hiding the details of what options are client, what are AWS Conf, and doing the wiring up of S3A stats interfaces to the AWS internals. This will break HBase's HBoss test suite, but I couldn't see a way to cleanly address this. I'll need to work on that code separately Change-Id: I4fb05f4a54eece8fac96abf8d6af9c889392a6c1
|
💔 -1 overall
This message was automatically generated. |
|
mehakmeet
reviewed
Mar 18, 2021
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestCustomSigner.java
Outdated
Show resolved
Hide resolved
| private static int invocationCount = 0; | ||
|
|
||
| private static AtomicInteger instantiationCount = new AtomicInteger(0); | ||
| private static AtomicInteger invocationCount = new AtomicInteger(0); |
...adoop-aws/src/test/java/org/apache/hadoop/fs/s3a/statistics/ITestAWSStatisticCollection.java
Outdated
Show resolved
Hide resolved
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
Outdated
Show resolved
Hide resolved
|
|
||
| /** | ||
| * Use the (newer) Builder SDK to create a an AWS S3 client. | ||
| * Use the Builder API to create a an AWS S3 client. |
...adoop-aws/src/test/java/org/apache/hadoop/fs/s3a/statistics/ITestAWSStatisticCollection.java
Outdated
Show resolved
Hide resolved
steveloughran
force-pushed
the
s3/HADOOP-13551-aws-metrics
branch
from
0923b23 to
66c0d43
Compare
Change-Id: I9ea831f5fec48ebcc7d08fe0e5d6918f8dfa2786
steveloughran
force-pushed
the
s3/HADOOP-13551-aws-metrics
branch
from
66c0d43 to
470d551
Compare
Change-Id: I6dd96b6ea4a69e1d11fca66c133ca5e880f57331
|
🎊 +1 overall
This message was automatically generated. |
This was referenced Mar 19, 2021
| @@ -226,31 +170,6 @@ protected static AmazonS3 configureAmazonS3Client(AmazonS3 s3, | |||
| throw new IllegalArgumentException(msg, e); | |||
| } | |||
| } | |||
There was a problem hiding this comment.
nit: a bit of commentry about path style access was good.
There's a lot of stuff in the docs about endpoints, including the exception we've been seeing here. Nothing else to add |
|
thanks! |
steveloughran
added a commit
to steveloughran/hadoop
that referenced
this pull request
Mar 24, 2021
Moves to the builder API for AWS S3 client creation, and offers a similar style of API to the S3A FileSystem and tests, hiding the details of which options are client, which are in AWS Conf, and doing the wiring up of S3A statistics interfaces to the AWS SDK internals. S3A Statistics, including IOStatistics, should now count throttling events handled in the AWS SDK itself. This patch restores endpoint determination by probes to US-East-1 if the client isn't configured with fs.s3a.endpoint. Explicitly setting the endpoint will save the cost of these probe HTTP requests. Contributed by Steve Loughran.
This was referenced Mar 24, 2021
asfgit
pushed a commit
that referenced
this pull request
Mar 25, 2021
Moves to the builder API for AWS S3 client creation, and offers a similar style of API to the S3A FileSystem and tests, hiding the details of which options are client, which are in AWS Conf, and doing the wiring up of S3A statistics interfaces to the AWS SDK internals. S3A Statistics, including IOStatistics, should now count throttling events handled in the AWS SDK itself. This patch restores endpoint determination by probes to US-East-1 if the client isn't configured with fs.s3a.endpoint. Explicitly setting the endpoint will save the cost of these probe HTTP requests. Contributed by Steve Loughran. Change-Id: Ifa6caa8ff56369ad30e4fd01a42bc74f7b8b3d6b
kiran-maturi
pushed a commit
to kiran-maturi/hadoop
that referenced
this pull request
Nov 24, 2021
Moves to the builder API for AWS S3 client creation, and offers a similar style of API to the S3A FileSystem and tests, hiding the details of which options are client, which are in AWS Conf, and doing the wiring up of S3A statistics interfaces to the AWS SDK internals. S3A Statistics, including IOStatistics, should now count throttling events handled in the AWS SDK itself. This patch restores endpoint determination by probes to US-East-1 if the client isn't configured with fs.s3a.endpoint. Explicitly setting the endpoint will save the cost of these probe HTTP requests. Contributed by Steve Loughran.
jojochuang
pushed a commit
to jojochuang/hadoop
that referenced
this pull request
May 23, 2023
Moves to the builder API for AWS S3 client creation, and offers a similar style of API to the S3A FileSystem and tests, hiding the details of which options are client, which are in AWS Conf, and doing the wiring up of S3A statistics interfaces to the AWS SDK internals. S3A Statistics, including IOStatistics, should now count throttling events handled in the AWS SDK itself. This patch restores endpoint determination by probes to US-East-1 if the client isn't configured with fs.s3a.endpoint. Explicitly setting the endpoint will save the cost of these probe HTTP requests. Contributed by Steve Loughran. Change-Id: Ifa6caa8ff56369ad30e4fd01a42bc74f7b8b3d6b (cherry picked from commit b32b08e1d208c45cf645d63a4630f34bfee5322e)
jojochuang
pushed a commit
to jojochuang/hadoop
that referenced
this pull request
May 23, 2023
This reverts commit 25c7250.
jojochuang
pushed a commit
to jojochuang/hadoop
that referenced
this pull request
May 23, 2023
Moves to the builder API for AWS S3 client creation, and offers a similar style of API to the S3A FileSystem and tests, hiding the details of which options are client, which are in AWS Conf, and doing the wiring up of S3A statistics interfaces to the AWS SDK internals. S3A Statistics, including IOStatistics, should now count throttling events handled in the AWS SDK itself. This patch restores endpoint determination by probes to US-East-1 if the client isn't configured with fs.s3a.endpoint. Explicitly setting the endpoint will save the cost of these probe HTTP requests. Contributed by Steve Loughran. Change-Id: Ifa6caa8ff56369ad30e4fd01a42bc74f7b8b3d6b
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This moves to a parameter-class and builder pattern for passing
in configuration parameters to the S3 client factory, so as to
stop adding any future parameters from breaking external implementations.
Extracted from HADOOP-17511 for isolated review/commit