Skip to content

HADOOP-18939. NPE in AWS v2 SDK RetryOnErrorCodeCondition.shouldRetry() #6193

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 17, 2023

Conversation

steveloughran
Copy link
Contributor

@steveloughran steveloughran commented Oct 16, 2023

Description of PR

Fixes MultiObjectDeleteException to completely fill in AWS SDK retrier requirements.

testing in progress

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

Contributed by Steve Loughran

Change-Id: Iaeb3707bd9716b1bdc53cc17611d903215760e91
@steveloughran
Copy link
Contributor Author

s3 london; failure is inevitably from region parser changes. will file

[ERROR] testWithoutRegionConfig(org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion)  Time elapsed: 2.872 s  <<< FAILURE!
org.junit.ComparisonFailure: [Incorrect number of calls made to get bucket region] expected:<[1]L> but was:<[0]L>
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion.assertRegionProbeCount(ITestS3AEndpointRegion.java:158)
        at org.apache.hadoop.fs.s3a.ITestS3AEndpointRegion.testWithoutRegionConfig(ITestS3AEndpointRegion.java:99)

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 25s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 33m 42s trunk passed
+1 💚 compile 0m 34s trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
+1 💚 compile 0m 29s trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚 checkstyle 0m 27s trunk passed
+1 💚 mvnsite 0m 34s trunk passed
+1 💚 javadoc 0m 25s trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 29s trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚 spotbugs 0m 52s trunk passed
+1 💚 shadedclient 20m 35s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 23s the patch passed
+1 💚 compile 0m 25s the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
+1 💚 javac 0m 25s the patch passed
+1 💚 compile 0m 21s the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚 javac 0m 21s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 16s the patch passed
+1 💚 mvnsite 0m 25s the patch passed
+1 💚 javadoc 0m 14s the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
+1 💚 javadoc 0m 20s the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
+1 💚 spotbugs 0m 47s the patch passed
+1 💚 shadedclient 20m 59s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 2m 33s /patch-unit-hadoop-tools_hadoop-aws.txt hadoop-aws in the patch passed.
+1 💚 asflicense 0m 29s The patch does not generate ASF License warnings.
88m 42s
Reason Tests
Failed junit tests hadoop.fs.s3a.impl.TestErrorTranslation
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6193/1/artifact/out/Dockerfile
GITHUB PR #6193
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux b1fbdb09f141 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 8dfcef2
Default Java Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6193/1/testReport/
Max. process+thread count 659 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6193/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@ahmarsuhail
Copy link
Contributor

that testWithoutRegionConfig is an intermittent failure I've seen before, never dug into it. with the latest patch region we remove that Store region probe completely, so don't need to worry about it once #6187 is merged.

Copy link
Contributor

@ahmarsuhail ahmarsuhail left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, code looks good.

But not super clear to me why we're doing this. The NPE will get fixed in the SDK. And I think RetryOnErrorCodeCondition.shouldRetry() will never be called with MultiObjectDeleteException as that is an exception we create in S3A?

@steveloughran
Copy link
Contributor Author

The NPE will get fixed in the SDK.

yes, but that's a future tense event. we are having problems with our deployments already

@steveloughran steveloughran merged commit e5eb404 into apache:trunk Oct 17, 2023
ahmarsuhail pushed a commit to ahmarsuhail/hadoop that referenced this pull request Nov 27, 2023
…() (apache#6193)


MultiObjectDeleteException to fill in the error details

See also: aws/aws-sdk-java-v2#4600

Contributed by Steve Loughran
ahmarsuhail pushed a commit to ahmarsuhail/hadoop that referenced this pull request Dec 5, 2023
…() (apache#6193)


MultiObjectDeleteException to fill in the error details

See also: aws/aws-sdk-java-v2#4600

Contributed by Steve Loughran
ahmarsuhail pushed a commit to ahmarsuhail/hadoop that referenced this pull request Dec 5, 2023
…() (apache#6193)


MultiObjectDeleteException to fill in the error details

See also: aws/aws-sdk-java-v2#4600

Contributed by Steve Loughran
jiajunmao pushed a commit to jiajunmao/hadoop-MLEC that referenced this pull request Feb 6, 2024
…() (apache#6193)


MultiObjectDeleteException to fill in the error details

See also: aws/aws-sdk-java-v2#4600

Contributed by Steve Loughran
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants