Skip to content

HADOOP-16080. hadoop-aws does not work with hadoop-client-api #2522

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Dec 12, 2020

Conversation

sunchao
Copy link
Member

@sunchao sunchao commented Dec 5, 2020

(This is a backport from #2510)

This does the following:

  • removes ListenableFuture as well as ListeningExecutorService from public interfaces, so that modules such as hadoop-aws and hadoop-aliyun can consume the class from hadoop-client-api without running into Guava conflicts.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 24m 48s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 2s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 6 new or modified test files.
_ branch-3.3 Compile Tests _
+0 🆗 mvndep 3m 57s Maven dependency ordering for branch
+1 💚 mvninstall 28m 33s branch-3.3 passed
+1 💚 compile 15m 51s branch-3.3 passed
+1 💚 checkstyle 2m 43s branch-3.3 passed
+1 💚 mvnsite 4m 28s branch-3.3 passed
+1 💚 shadedclient 23m 7s branch has no errors when building and testing our client artifacts.
-1 ❌ javadoc 0m 40s hadoop-aws in branch-3.3 failed.
+0 🆗 spotbugs 1m 7s Used deprecated FindBugs config; considering switching to SpotBugs.
-1 ❌ findbugs 0m 50s hadoop-cloud-storage-project/hadoop-cos in branch-3.3 has 1 extant findbugs warnings.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 26s Maven dependency ordering for patch
+1 💚 mvninstall 2m 34s the patch passed
+1 💚 compile 14m 48s the patch passed
+1 💚 javac 14m 48s the patch passed
-0 ⚠️ checkstyle 2m 42s root: The patch generated 1 new + 101 unchanged - 0 fixed = 102 total (was 101)
+1 💚 mvnsite 4m 26s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 14m 42s patch has no errors when building and testing our client artifacts.
-1 ❌ javadoc 0m 42s hadoop-aws in the patch failed.
-1 ❌ findbugs 1m 23s hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
_ Other Tests _
+1 💚 unit 9m 33s hadoop-common in the patch passed.
+1 💚 unit 1m 36s hadoop-aws in the patch passed.
+1 💚 unit 1m 36s hadoop-azure in the patch passed.
+1 💚 unit 0m 37s hadoop-aliyun in the patch passed.
+1 💚 unit 0m 37s hadoop-cos in the patch passed.
+1 💚 asflicense 0m 55s The patch does not generate ASF License warnings.
176m 31s
Reason Tests
FindBugs module:hadoop-tools/hadoop-aws
Exceptional return value of java.util.concurrent.ExecutorService.submit(Callable) ignored in org.apache.hadoop.fs.s3a.impl.StoreContext.submit(CompletableFuture, Callable) At StoreContext.java:ignored in org.apache.hadoop.fs.s3a.impl.StoreContext.submit(CompletableFuture, Callable) At StoreContext.java:[line 385]
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/1/artifact/out/Dockerfile
GITHUB PR #2522
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux fa52ec13fcb6 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / 4628647
Default Java Private Build-1.8.0_275-8u275-b01-0ubuntu1~16.04-b01
javadoc https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/1/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws.txt
findbugs https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/1/artifact/out/branch-findbugs-hadoop-cloud-storage-project_hadoop-cos-warnings.html
checkstyle https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/1/artifact/out/diff-checkstyle-root.txt
javadoc https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/1/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws.txt
findbugs https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/1/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/1/testReport/
Max. process+thread count 1481 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure hadoop-tools/hadoop-aliyun hadoop-cloud-storage-project/hadoop-cos U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/1/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.*;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wildcard should be avoided.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops will do - I forgot to set this up on my new laptop

@aajisaka
Copy link
Member

aajisaka commented Dec 7, 2020

replaces error message template usages in checkArgument/checkState/checkNotNull with String.format. The former is generally not available in Guava version < 20 so this is to eliminate potential conflicts.

Rethinking this, using String.format is slower than the template usages because it creates a string error message before checking the condition. Now I think we need to create our internal checkArgument/checkState/checkNotNull functions and it can be done in a separate JIRA.

@sunchao
Copy link
Member Author

sunchao commented Dec 7, 2020

@aajisaka yes whatever done here is only a short-term fix to make hadoop-aws work with hadoop-client-api (it is more urgent for 3.2.2 since it is already in the process of release). Eventually we should do something similar to #2134 .

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 32m 5s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 6 new or modified test files.
_ branch-3.3 Compile Tests _
+0 🆗 mvndep 3m 57s Maven dependency ordering for branch
+1 💚 mvninstall 35m 4s branch-3.3 passed
+1 💚 compile 19m 39s branch-3.3 passed
+1 💚 checkstyle 2m 39s branch-3.3 passed
+1 💚 mvnsite 4m 14s branch-3.3 passed
+1 💚 shadedclient 25m 5s branch has no errors when building and testing our client artifacts.
-1 ❌ javadoc 0m 39s hadoop-aws in branch-3.3 failed.
+0 🆗 spotbugs 1m 3s Used deprecated FindBugs config; considering switching to SpotBugs.
-1 ❌ findbugs 0m 47s hadoop-cloud-storage-project/hadoop-cos in branch-3.3 has 1 extant findbugs warnings.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 28s Maven dependency ordering for patch
+1 💚 mvninstall 2m 41s the patch passed
+1 💚 compile 14m 50s the patch passed
+1 💚 javac 14m 50s the patch passed
-0 ⚠️ checkstyle 2m 35s root: The patch generated 1 new + 101 unchanged - 0 fixed = 102 total (was 101)
+1 💚 mvnsite 4m 24s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 14m 47s patch has no errors when building and testing our client artifacts.
-1 ❌ javadoc 0m 41s hadoop-aws in the patch failed.
-1 ❌ findbugs 1m 23s hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
_ Other Tests _
+1 💚 unit 9m 49s hadoop-common in the patch passed.
+1 💚 unit 1m 42s hadoop-aws in the patch passed.
+1 💚 unit 1m 38s hadoop-azure in the patch passed.
+1 💚 unit 0m 30s hadoop-aliyun in the patch passed.
+1 💚 unit 0m 34s hadoop-cos in the patch passed.
+1 💚 asflicense 0m 50s The patch does not generate ASF License warnings.
196m 11s
Reason Tests
FindBugs module:hadoop-tools/hadoop-aws
Exceptional return value of java.util.concurrent.ExecutorService.submit(Callable) ignored in org.apache.hadoop.fs.s3a.impl.StoreContext.submit(CompletableFuture, Callable) At StoreContext.java:ignored in org.apache.hadoop.fs.s3a.impl.StoreContext.submit(CompletableFuture, Callable) At StoreContext.java:[line 385]
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/2/artifact/out/Dockerfile
GITHUB PR #2522
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 2798fb5dc3b8 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / cb2dce3
Default Java Private Build-1.8.0_275-8u275-b01-0ubuntu1~16.04-b01
javadoc https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/2/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws.txt
findbugs https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/2/artifact/out/branch-findbugs-hadoop-cloud-storage-project_hadoop-cos-warnings.html
checkstyle https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/2/artifact/out/diff-checkstyle-root.txt
javadoc https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/2/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws.txt
findbugs https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/2/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/2/testReport/
Max. process+thread count 1394 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure hadoop-tools/hadoop-aliyun hadoop-cloud-storage-project/hadoop-cos U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/2/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@sunchao
Copy link
Member Author

sunchao commented Dec 9, 2020

@aajisaka I've removed all the changes on Preconditions as #2505 should help to solve the issue once it's merged. Can you take another look? thanks.

@aajisaka
Copy link
Member

aajisaka commented Dec 9, 2020

@sunchao Would you fix the findbugs warning or ignore it by adding an entry to the findbugs-exclude.xml? I'm +1 if that is addressed.

Exceptional return value of java.util.concurrent.ExecutorService.submit(Callable) ignored in org.apache.hadoop.fs.s3a.impl.StoreContext.submit(CompletableFuture, Callable) At StoreContext.java:ignored in org.apache.hadoop.fs.s3a.impl.StoreContext.submit(CompletableFuture, Callable) At StoreContext.java:[line 385]

@sunchao
Copy link
Member Author

sunchao commented Dec 9, 2020

@aajisaka sure added to findbugs-exclude.xml - note that this PR doesn't touch anything on the method though.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 39s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 6 new or modified test files.
_ branch-3.3 Compile Tests _
+0 🆗 mvndep 4m 3s Maven dependency ordering for branch
+1 💚 mvninstall 28m 23s branch-3.3 passed
+1 💚 compile 15m 34s branch-3.3 passed
+1 💚 checkstyle 2m 47s branch-3.3 passed
+1 💚 mvnsite 4m 28s branch-3.3 passed
+1 💚 shadedclient 23m 5s branch has no errors when building and testing our client artifacts.
-1 ❌ javadoc 0m 43s hadoop-aws in branch-3.3 failed.
+0 🆗 spotbugs 1m 7s Used deprecated FindBugs config; considering switching to SpotBugs.
-1 ❌ findbugs 0m 51s hadoop-cloud-storage-project/hadoop-cos in branch-3.3 has 1 extant findbugs warnings.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 24s Maven dependency ordering for patch
+1 💚 mvninstall 2m 37s the patch passed
+1 💚 compile 14m 53s the patch passed
+1 💚 javac 14m 53s the patch passed
-0 ⚠️ checkstyle 2m 42s root: The patch generated 1 new + 101 unchanged - 0 fixed = 102 total (was 101)
+1 💚 mvnsite 4m 25s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 1s The patch has no ill-formed XML file.
+1 💚 shadedclient 14m 47s patch has no errors when building and testing our client artifacts.
-1 ❌ javadoc 0m 42s hadoop-aws in the patch failed.
-1 ❌ findbugs 1m 21s hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
_ Other Tests _
-1 ❌ unit 9m 31s hadoop-common in the patch passed.
+1 💚 unit 1m 41s hadoop-aws in the patch passed.
+1 💚 unit 1m 37s hadoop-azure in the patch passed.
+1 💚 unit 0m 37s hadoop-aliyun in the patch passed.
+1 💚 unit 0m 37s hadoop-cos in the patch passed.
+1 💚 asflicense 0m 55s The patch does not generate ASF License warnings.
152m 26s
Reason Tests
FindBugs module:hadoop-tools/hadoop-aws
Exceptional return value of java.util.concurrent.ExecutorService.submit(Callable) ignored in org.apache.hadoop.fs.s3a.impl.StoreContext.submit(CompletableFuture, Callable) At StoreContext.java:ignored in org.apache.hadoop.fs.s3a.impl.StoreContext.submit(CompletableFuture, Callable) At StoreContext.java:[line 385]
Failed junit tests hadoop.security.TestGroupsCaching
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/3/artifact/out/Dockerfile
GITHUB PR #2522
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname Linux 8368c601dba6 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / f04a9df
Default Java Private Build-1.8.0_275-8u275-b01-0ubuntu1~16.04-b01
javadoc https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/3/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws.txt
findbugs https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/3/artifact/out/branch-findbugs-hadoop-cloud-storage-project_hadoop-cos-warnings.html
checkstyle https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/3/artifact/out/diff-checkstyle-root.txt
javadoc https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/3/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws.txt
findbugs https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/3/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html
unit https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/3/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/3/testReport/
Max. process+thread count 1483 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure hadoop-tools/hadoop-aliyun hadoop-cloud-storage-project/hadoop-cos U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/3/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@sunchao sunchao force-pushed the HADOOP-16080-branch-3.3 branch from eb08721 to 8600f19 Compare December 10, 2020 06:47
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 37s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ branch-3.3 Compile Tests _
+0 🆗 mvndep 3m 59s Maven dependency ordering for branch
+1 💚 mvninstall 28m 42s branch-3.3 passed
+1 💚 compile 15m 46s branch-3.3 passed
+1 💚 checkstyle 2m 37s branch-3.3 passed
+1 💚 mvnsite 3m 56s branch-3.3 passed
+1 💚 shadedclient 25m 54s branch has no errors when building and testing our client artifacts.
-1 ❌ javadoc 0m 37s hadoop-aws in branch-3.3 failed.
+0 🆗 spotbugs 1m 25s Used deprecated FindBugs config; considering switching to SpotBugs.
-1 ❌ findbugs 0m 49s hadoop-cloud-storage-project/hadoop-cos in branch-3.3 has 1 extant findbugs warnings.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 27s Maven dependency ordering for patch
+1 💚 mvninstall 2m 33s the patch passed
+1 💚 compile 19m 29s the patch passed
+1 💚 javac 19m 29s the patch passed
-0 ⚠️ checkstyle 3m 6s root: The patch generated 1 new + 46 unchanged - 0 fixed = 47 total (was 46)
+1 💚 mvnsite 3m 16s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 1s The patch has no ill-formed XML file.
+1 💚 shadedclient 15m 6s patch has no errors when building and testing our client artifacts.
-1 ❌ javadoc 0m 39s hadoop-aws in the patch failed.
+1 💚 findbugs 5m 28s the patch passed
_ Other Tests _
+1 💚 unit 9m 29s hadoop-common in the patch passed.
+1 💚 unit 1m 37s hadoop-aws in the patch passed.
+1 💚 unit 0m 33s hadoop-aliyun in the patch passed.
+1 💚 unit 0m 35s hadoop-cos in the patch passed.
+1 💚 asflicense 0m 51s The patch does not generate ASF License warnings.
153m 54s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/4/artifact/out/Dockerfile
GITHUB PR #2522
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname Linux 0b90c5f955c2 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / b84bc63
Default Java Private Build-1.8.0_275-8u275-b01-0ubuntu1~16.04-b01
javadoc https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/4/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws.txt
findbugs https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/4/artifact/out/branch-findbugs-hadoop-cloud-storage-project_hadoop-cos-warnings.html
checkstyle https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/4/artifact/out/diff-checkstyle-root.txt
javadoc https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/4/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws.txt
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/4/testReport/
Max. process+thread count 1605 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-aliyun hadoop-cloud-storage-project/hadoop-cos U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2522/4/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Copy link
Member

@aajisaka aajisaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, +1. Thank you @sunchao

@aajisaka
Copy link
Member

Backported https://issues.apache.org/jira/browse/HADOOP-17186 to fix the javadoc error in hadoop-aws module.

@aajisaka
Copy link
Member

Note: In branch-3.3, there is a findbugs warning in hadoop-cloud-storage-project/hadoop-cos. I'm going to cherry-pick HADOOP-17138 in #2538 to fix the warning.

@sunchao
Copy link
Member Author

sunchao commented Dec 10, 2020

Thanks @aajisaka !

@sunchao sunchao merged commit 81e533d into apache:branch-3.3 Dec 12, 2020
sunchao added a commit to sunchao/hadoop that referenced this pull request Jan 9, 2021
steveloughran pushed a commit that referenced this pull request Mar 9, 2021
kiran-maturi pushed a commit to kiran-maturi/hadoop that referenced this pull request Nov 24, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants