HADOOP-14661. Add S3 requester pays bucket support to S3A

[dannycjones]

Description of PR

This change introduces support for "requester pays" S3 buckets (HADOOP-14661).

There was already some discussion and patches made available (e.g. #250), although the code base has changed a bit since then.

In summary:

• Add some information to the main S3A documentation
• Add example of error and how to fix/configure to troubleshooting doc
• Add options to configure requester pays in S3A (fs.s3a.requester-pays.enabled) when creating S3A filesystem, which uses S3ClientFactory to create a new S3Client also taking this option
• Add the required header inside S3ClientFactory if option was configured
• Remove unused configuration options for requester pays in RequestFactoryImpl
• Add new integration test which talks to publicly available usgs-landsat bucket (new version, not old) which is configured to require requester pays ('usgs-landsat' on Registry of Open Data on AWS).
• Add information on how to configure the new tests in testing.md

How was this patch tested?

I created an EC2 instance and an S3 Bucket in eu-west-1 and ran the tests against the global S3 endpoint.

mvn -Dparallel-tests -DtestsThreadCount=16 clean verify

Note, the request pays specific tests introduced do not use the developer supplied bucket.

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[monthonk]

Looks good to me.

[monthonk]

Approved

[dannycjones]

@mukund-thakur would you be able to review this PR?

[mukund-thakur]

Sure will check early next week.

[mukund-thakur]

CC @steveloughran @mehakmeet

[mukund-thakur]

Overall looks good. Some minor comments.

[steveloughran]

thanks for this, sorry I've been unreliable at reviewing.

if the SDK does now attach the requester pays header everywhere then we don't need the stuff in the request factory...that was in as the discussion implied SDK coverage was incomplete.

Test-wise, made some comments. I think it also might be good to try and do some more API calls to make sure the requests are tagged, listing in particular

[dannycjones]

Thank you both for the comments, I have just pushed a few changes.

I also removed unused configuration from the RequestFactoryImpl - I am assuming since its under .impl package, it's fair game to remove without deprecation.

[steveloughran]

I am assuming since its under .impl package, it's fair game to remove without deprecation.

yes. though we do regularly found out where that's not always true (s3 client factory, store context). how do we find out? change things and see what breaks, then add comments in the javadocs to warn others

[steveloughran]

i like the tests, much cleaner now. made some minor suggestions there.

if you can do those and cover how to change/disable the tests in testing.md, this is ready to go in

[steveloughran]

+1
this is good to go. one little detail, which is those line endings in the markdown.

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md:1638:S3A supports buckets with 
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md:1639:[Requester Pays](https://docs.aws.amazon.com/AmazonS3/latest/userguide/RequesterPaysBuckets.html) 
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md:1640:enabled. When a bucket is configured with requester pays, the requester must cover 
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md:1643:For requests to be successful, the S3 client must acknowledge that they will pay 
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md:598:By default, the requester pays tests will look for a bucket that exists on Amazon S3 
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md:602:The test only requires an object of at least a few bytes in order 
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md:612:If the endpoint does not support requester pays, you can also disable the tests by configuring 

can you fix these as the github merge button doesn't do it for us?

[dannycjones]

@steveloughran I caught those after I pushed most of the other fixes from the last feedback and pushed a second commit ae8e812.

There's two Yetus comments, latest one seems happy. Are we good to merge?

[steveloughran]

+1, I am happy

[steveloughran]

merged to trunk, will cherrypick to 3.3 locally, retest and if all is good merge there too

[dannycjones]

Thanks Steve, Mukund, and Monthon for the reviews!