Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HADOOP-16080. hadoop-aws does not work with hadoop-client-api #2510

Merged
merged 4 commits into from
Dec 4, 2020

Conversation

sunchao
Copy link
Member

@sunchao sunchao commented Dec 2, 2020

This does the following:

  • removes ListenableFuture as well as ListeningExecutorService from SemaphoredDelegatingExecutor, so that modules such as hadoop-aws and hadoop-aliyun can consume the class from hadoop-client-api without running into Guava conflicts.
  • replaces error message template usages in checkArgument/checkState/checkNotNull with String.format. The former is generally not available in Guava version < 20 so this is to eliminate potential conflicts.

@sunchao sunchao marked this pull request as draft December 2, 2020 01:18
@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 16m 8s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ branch-3.2 Compile Tests _
+0 🆗 mvndep 3m 52s Maven dependency ordering for branch
+1 💚 mvninstall 25m 54s branch-3.2 passed
+1 💚 compile 15m 31s branch-3.2 passed
+1 💚 checkstyle 2m 39s branch-3.2 passed
+1 💚 mvnsite 2m 23s branch-3.2 passed
+1 💚 shadedclient 19m 23s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 2m 1s branch-3.2 passed
+0 🆗 spotbugs 0m 57s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 3m 39s branch-3.2 passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 19s Maven dependency ordering for patch
+1 💚 mvninstall 1m 45s the patch passed
+1 💚 compile 14m 30s the patch passed
+1 💚 javac 14m 30s the patch passed
+1 💚 checkstyle 2m 36s the patch passed
+1 💚 mvnsite 2m 26s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 13m 28s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 1m 59s the patch passed
+1 💚 findbugs 3m 58s the patch passed
_ Other Tests _
+1 💚 unit 9m 14s hadoop-common in the patch passed.
+1 💚 unit 4m 44s hadoop-aws in the patch passed.
+1 💚 unit 0m 26s hadoop-aliyun in the patch passed.
+1 💚 asflicense 0m 44s The patch does not generate ASF License warnings.
147m 22s
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2510/1/artifact/out/Dockerfile
GITHUB PR #2510
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 4312a09ef128 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.2 / 625f85f
Default Java Private Build-1.8.0_275-8u275-b01-0ubuntu1~16.04-b01
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2510/1/testReport/
Max. process+thread count 1389 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-aliyun U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2510/1/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 35s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ branch-3.2 Compile Tests _
+0 🆗 mvndep 3m 51s Maven dependency ordering for branch
+1 💚 mvninstall 25m 56s branch-3.2 passed
+1 💚 compile 15m 19s branch-3.2 passed
+1 💚 checkstyle 2m 40s branch-3.2 passed
+1 💚 mvnsite 2m 27s branch-3.2 passed
+1 💚 shadedclient 19m 37s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 2m 5s branch-3.2 passed
+0 🆗 spotbugs 0m 56s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 3m 32s branch-3.2 passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 20s Maven dependency ordering for patch
+1 💚 mvninstall 1m 46s the patch passed
+1 💚 compile 15m 18s the patch passed
+1 💚 javac 15m 18s the patch passed
+1 💚 checkstyle 3m 27s the patch passed
+1 💚 mvnsite 2m 47s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 15m 43s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 2m 27s the patch passed
+1 💚 findbugs 5m 5s the patch passed
_ Other Tests _
-1 ❌ unit 10m 14s hadoop-common in the patch failed.
+1 💚 unit 5m 5s hadoop-aws in the patch passed.
+1 💚 unit 0m 35s hadoop-aliyun in the patch passed.
+1 💚 asflicense 1m 26s The patch does not generate ASF License warnings.
141m 26s
Reason Tests
Failed junit tests hadoop.security.TestFixKerberosTicketOrder
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2510/2/artifact/out/Dockerfile
GITHUB PR #2510
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 23f531e1c395 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.2 / 625f85f
Default Java Private Build-1.8.0_275-8u275-b01-0ubuntu1~16.04-b01
unit https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2510/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2510/2/testReport/
Max. process+thread count 1509 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-aliyun U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2510/2/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Copy link
Member

@aajisaka aajisaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, +1

@aajisaka
Copy link
Member

aajisaka commented Dec 2, 2020

BTW, I think trunk and branch-3.3 can be updated as well in a separate JIRA to avoid the usage of guava as possible.

@sunchao
Copy link
Member Author

sunchao commented Dec 2, 2020

Thanks @aajisaka ! yes I do plan to apply this to trunk and branch-3.3. I hope this aligns with the work in HADOOP-17098 (cc @amahussein) which is more general.

cc @steveloughran too

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 15s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ branch-3.2 Compile Tests _
+0 🆗 mvndep 3m 52s Maven dependency ordering for branch
+1 💚 mvninstall 25m 49s branch-3.2 passed
+1 💚 compile 15m 20s branch-3.2 passed
+1 💚 checkstyle 2m 36s branch-3.2 passed
+1 💚 mvnsite 2m 25s branch-3.2 passed
+1 💚 shadedclient 19m 37s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 2m 2s branch-3.2 passed
+0 🆗 spotbugs 0m 58s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 3m 35s branch-3.2 passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 19s Maven dependency ordering for patch
+1 💚 mvninstall 1m 47s the patch passed
+1 💚 compile 14m 22s the patch passed
+1 💚 javac 14m 22s the patch passed
-0 ⚠️ checkstyle 2m 33s root: The patch generated 1 new + 56 unchanged - 0 fixed = 57 total (was 56)
+1 💚 mvnsite 2m 24s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 13m 40s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 1m 58s the patch passed
+1 💚 findbugs 3m 58s the patch passed
_ Other Tests _
+1 💚 unit 10m 0s hadoop-common in the patch passed.
+1 💚 unit 4m 39s hadoop-aws in the patch passed.
+1 💚 unit 0m 25s hadoop-aliyun in the patch passed.
+1 💚 asflicense 0m 45s The patch does not generate ASF License warnings.
133m 8s
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2510/3/artifact/out/Dockerfile
GITHUB PR #2510
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 20f6dd67b5a2 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.2 / 1184284
Default Java Private Build-1.8.0_275-8u275-b01-0ubuntu1~16.04-b01
checkstyle https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2510/3/artifact/out/diff-checkstyle-root.txt
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2510/3/testReport/
Max. process+thread count 1346 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws hadoop-tools/hadoop-aliyun U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2510/3/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@sunchao sunchao merged commit d82b0cc into apache:branch-3.2 Dec 4, 2020
@sunchao
Copy link
Member Author

sunchao commented Dec 4, 2020

Merged to branch-3.2. I'll create PRs for branch-3.1 and trunk later. Thanks again @aajisaka for the review.

@apache apache deleted a comment from hadoop-yetus Dec 4, 2020
asfgit pushed a commit that referenced this pull request Dec 4, 2020
sunchao added a commit to sunchao/hadoop that referenced this pull request Dec 10, 2020
@steveloughran
Copy link
Contributor

FWIW that checkArgument(varags) does exist in older versions; what's gone wrong is that google added specific overloads for certain non-varargs variant, e.g (String, int) so that stuff built against the newer version isn't going to link. Adding a String.format does make that arg checking a bit more expensive, as it's being calculated even on the correct sequence. We are moving to our own precondition checker because of the guava pain; we should switch to that

@sunchao
Copy link
Member Author

sunchao commented Dec 12, 2020

Yes this is a hot fix to make it work with 3.2.2 but totally agree that we should avoid the perf hit perhaps through #2505. In 3.3.x and up this is no longer an issue since we've switched to the third party guava library.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants