-
Notifications
You must be signed in to change notification settings - Fork 9.1k
HADOOP-16080. hadoop-aws does not work with hadoop-client-api #2575
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
This comment has been minimized.
This comment has been minimized.
could you review this @aajisaka? no conflict in the backport. cc @steveloughran too. |
#2522 should have gone into trunk first there should be nothing in an older branch which is not in trunk |
@steveloughran my bad. Should've done this in a proper way. |
79db74e
to
ec20769
Compare
Usual due diligence query: which s3 endpoint did you run the integration tests against? (I'll expect some tests failures there from HADOOP-16380 stabilisation; if you don't find them I'd be worried about your test setup...they won't be blockers) |
@steveloughran eh I only tested this in Spark (verified that the failure here was fixed while was reproducible w/o the PR) using a S3A end point of my own. I can run the integration tests also - are the steps here? |
Tried the steps above - most tests succeeded but I got a bunch of failures related to Dynamo DB table not found:
I'm confused since I didn't have neither |
bq. I got a bunch of failures related to Dynamo DB table not found: some of the tests are parameterized to do test runs with/without dynamoDB. They shouldn't be run if the -Ddynamo option wasn't set, but what has inevitably happened is that regressions into the test runs have crept in and we've not noticed. if you are testing against a non-AWS endpoint we should skip tests related to that, IAM roles/session tokens and encryption File a JIRA about it, tell us which tests are failing, etc... |
BTW, does this mean your initial PR went in without running the ITests? Not good. We like to have a strict "no tests, no review" policy for the hadoop-aws, hadoop-azure branches. We can't give yetus the credentials for test runs, and people need to be set up to debug why their patches don't work anyway. You managed to dodge a bit of the diligence requirements there. Afraid I'm being strict now. If tests fail, list which ones did and we'll see if they are new/existing intermittent ones/recent regressions. #2594 is trying to fix some which have now surfaced in some test configs |
I didn't specify the
I'm testing against my own S3A endpoint "s3a://sunchao/" which is in us-west-1 and I just followed the doc to setup
Unfortunately no ... sorry I was not aware of the test steps here (first time contributing to hadoop-aws). I'll try to do some remedy in this PR. Test failures I got:
And seems most of the failures are due to error like the following:
Not sure if I missed some steps in my test setup. |
Ignoring the s3guard/ddb ones (we're clearly still trying to run some when those tests aren't enabled), you are going to be seeing the failures covered in https://issues.apache.org/jira/browse/HADOOP-17451 / #2594 . I'll get that ready for merging today...mostly it's some of the tests using metrics being brittle to how they are executed |
something went wrong with yetus; try a rebase and forced push |
…buted by Chao Sun (apache#2522)
ec20769
to
c22d19f
Compare
🎊 +1 overall
This message was automatically generated. |
I've been running on trunk for a while, everything is happy there. Going to +1 and merge this backport |
thanks @steveloughran ! |
…e#2522) Contributed by Chao Sun. (Cherry-picked via PR apache#2575)
This is a backport of #2522 to trunk