-
Notifications
You must be signed in to change notification settings - Fork 8.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HADOOP-17705. S3A to add Config to set AWS region #3020
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fix those yetus failures.
LGTM, will wait for yetus to become green. |
+1 pending docs. |
in docs, if we have a stack trace of the error message you get before this patch is in, add it to the troubleshooting doc with a reference to the option as well |
LGTM; just need to strip out the EOLs for a merge from the web UI. |
🎊 +1 overall
This message was automatically generated. |
merged to trunk, backporting if the new test runs OK there |
The option `fs.s3a.endpoint.region` can be used to explicitly set the AWS region of a bucket. This is needed when using AWS Private Link, as the region cannot be automatically determined. Contributed by Mehakmeet Singh Change-Id: I4b52f85d7af0ddd56b2b0505ac0124d4fcc67ca0
The option `fs.s3a.endpoint.region` can be used to explicitly set the AWS region of a bucket. This is needed when using AWS Private Link, as the region cannot be automatically determined. Contributed by Mehakmeet Singh
The option `fs.s3a.endpoint.region` can be used to explicitly set the AWS region of a bucket. This is needed when using AWS Private Link, as the region cannot be automatically determined. Contributed by Mehakmeet Singh
The option `fs.s3a.endpoint.region` can be used to explicitly set the AWS region of a bucket. This is needed when using AWS Private Link, as the region cannot be automatically determined. Contributed by Mehakmeet Singh
…#3020) The option `fs.s3a.endpoint.region` can be used to explicitly set the AWS region of a bucket. This is needed when using AWS Private Link, as the region cannot be automatically determined. Contributed by Mehakmeet Singh Change-Id: If9b1b1d8ea3556ed4dfb2d164a8a6fa477dfc36d (cherry picked from commit 27d27beb319bcbee7d89514b6cd7ba88e325019b)
…apache#3020)" This reverts commit 203b7c2.
…#3020) The option `fs.s3a.endpoint.region` can be used to explicitly set the AWS region of a bucket. This is needed when using AWS Private Link, as the region cannot be automatically determined. Contributed by Mehakmeet Singh *Important* This has proven to be a fairly brittle change. Consult the CDPD-26441 JIRA for details. Test environments with the AWS CLI or run in EC2 do not pick up regressions, so it is unlikely that test runs will identify major issues which surfaced in production. When backported it MUST be accompanied by: CDPD-27264. HADOOP-17771. S3AFS creation fails "Unable to find a region via the region provider chain." This patch depends on CDPD-21205, which breaks the HBoss build/test. Read the instructions there when cherrypicking. Change-Id: If9b1b1d8ea3556ed4dfb2d164a8a6fa477dfc36d
…on via the region provider chain." (apache#3133) This addresses the regression in Hadoop 3.3.1 where if no S3 endpoint is set in fs.s3a.endpoint, S3A filesystem creation may fail on non-EC2 deployments, depending on the local host environment setup. * If fs.s3a.endpoint is empty/null, and fs.s3a.endpoint.region is null, the region is set to "us-east-1". * If fs.s3a.endpoint.region is explicitly set to "" then the client falls back to the SDK region resolution chain; this works on EC2 * Details in troubleshooting.md, including a workaround for Hadoop-3.3.1+ * Also contains some minor restructuring of troubleshooting.md * uses pre-Auditing LogExactlyOnce import, so doesn't depend on that patch. Contributed by Steve Loughran. This is a critical follow on patch to CDPD-26441. HADOOP-17705. S3A to add Config to set AWS region (apache#3020) Both patches must be included Change-Id: Icca928e1752423d68591508c360ff6434997fb64
Tested by: mvn clean verify -Dparallel-tests -Dscale
Region: ap-south-1
Test:
Timeout and intermittent failures.
CC: @steveloughran @mukund-thakur