HADOOP-16341. ShutDownHookManager: Regressed performance on Hook removals after HADOOP-15679 #1086

zeroflag · 2019-07-12T14:44:59Z

Continuation of #940. Taking it over from @t3rmin4t0r.

…vals after HADOOP-15679

zeroflag · 2019-07-12T15:36:55Z

@steveloughran could you review it? It's the same as #940 but with the fix for the flaky test.

hadoop-yetus · 2019-07-12T16:38:12Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	72	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	1288	trunk passed
+1	compile	1163	trunk passed
+1	checkstyle	47	trunk passed
+1	mvnsite	92	trunk passed
+1	shadedclient	896	branch has no errors when building and testing our client artifacts.
+1	javadoc	64	trunk passed
0	spotbugs	129	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	126	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	56	the patch passed
+1	compile	1140	the patch passed
+1	javac	1140	the patch passed
-0	checkstyle	46	hadoop-common-project/hadoop-common: The patch generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2)
+1	mvnsite	88	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	745	patch has no errors when building and testing our client artifacts.
+1	javadoc	62	the patch passed
+1	findbugs	140	the patch passed
		_ Other Tests _
-1	unit	592	hadoop-common in the patch failed.
+1	asflicense	49	The patch does not generate ASF License warnings.
		6735

Reason	Tests
Failed junit tests	hadoop.util.TestShutdownHookManager

Subsystem	Report/Notes
Docker	Client=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1086/1/artifact/out/Dockerfile
GITHUB PR	#1086
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 1c6568919351 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `190e434`
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1086/1/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1086/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1086/1/testReport/
Max. process+thread count	1728 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1086/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-07-12T16:41:32Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	83	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	1252	trunk passed
+1	compile	1158	trunk passed
+1	checkstyle	45	trunk passed
+1	mvnsite	85	trunk passed
+1	shadedclient	888	branch has no errors when building and testing our client artifacts.
+1	javadoc	69	trunk passed
0	spotbugs	149	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	146	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	70	the patch passed
+1	compile	1110	the patch passed
+1	javac	1110	the patch passed
-0	checkstyle	44	hadoop-common-project/hadoop-common: The patch generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2)
+1	mvnsite	80	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	726	patch has no errors when building and testing our client artifacts.
+1	javadoc	67	the patch passed
+1	findbugs	160	the patch passed
		_ Other Tests _
-1	unit	572	hadoop-common in the patch failed.
+1	asflicense	45	The patch does not generate ASF License warnings.
		6682

Reason	Tests
Failed junit tests	hadoop.util.TestShutdownHookManager

Subsystem	Report/Notes
Docker	Client=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1086/2/artifact/out/Dockerfile
GITHUB PR	#1086
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 7c540dd37401 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `190e434`
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1086/2/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1086/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1086/2/testReport/
Max. process+thread count	1668 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1086/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

…vals after HADOOP-15679

hadoop-yetus · 2019-07-13T15:59:04Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	105	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	1179	trunk passed
+1	compile	1351	trunk passed
+1	checkstyle	48	trunk passed
+1	mvnsite	75	trunk passed
+1	shadedclient	817	branch has no errors when building and testing our client artifacts.
+1	javadoc	66	trunk passed
0	spotbugs	129	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	126	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	59	the patch passed
+1	compile	1248	the patch passed
+1	javac	1248	the patch passed
-0	checkstyle	43	hadoop-common-project/hadoop-common: The patch generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2)
+1	mvnsite	74	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	688	patch has no errors when building and testing our client artifacts.
+1	javadoc	62	the patch passed
+1	findbugs	131	the patch passed
		_ Other Tests _
+1	unit	539	hadoop-common in the patch passed.
+1	asflicense	42	The patch does not generate ASF License warnings.
		6698

Subsystem	Report/Notes
Docker	Client=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1086/3/artifact/out/Dockerfile
GITHUB PR	#1086
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 6325d4029d38 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `4a70a0d`
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1086/3/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1086/3/testReport/
Max. process+thread count	1391 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1086/3/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

steveloughran

makes sense: you've made it testable and done the test. As you are now about to learn: my expectation from all unit tests are that every assert needs meaningful messages.assertEquals is more self-explanatory, but I do want messages in the others

...mmon-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestShutdownHookManager.java

…vals after HADOOP-15679

hadoop-yetus · 2019-07-15T14:04:21Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	528	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	1127	trunk passed
+1	compile	1059	trunk passed
+1	checkstyle	39	trunk passed
+1	mvnsite	73	trunk passed
+1	shadedclient	732	branch has no errors when building and testing our client artifacts.
+1	javadoc	55	trunk passed
0	spotbugs	120	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	119	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	46	the patch passed
+1	compile	979	the patch passed
+1	javac	979	the patch passed
-0	checkstyle	39	hadoop-common-project/hadoop-common: The patch generated 3 new + 2 unchanged - 0 fixed = 5 total (was 2)
+1	mvnsite	72	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	629	patch has no errors when building and testing our client artifacts.
+1	javadoc	57	the patch passed
+1	findbugs	128	the patch passed
		_ Other Tests _
-1	unit	527	hadoop-common in the patch failed.
+1	asflicense	45	The patch does not generate ASF License warnings.
		6326

Reason	Tests
Failed junit tests	hadoop.ha.TestZKFailoverControllerStress

Subsystem	Report/Notes
Docker	Client=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1086/4/artifact/out/Dockerfile
GITHUB PR	#1086
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 88b1e5f7794c 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `5446308`
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1086/4/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1086/4/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1086/4/testReport/
Max. process+thread count	1416 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1086/4/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

zeroflag · 2019-07-15T15:35:09Z

The failure in TestZKFailoverControllerStress is not related to this change.

zeroflag · 2019-07-15T20:46:21Z

@steveloughran could you review it again and merge it if everything is ok?

cc @t3rmin4t0r.

steveloughran · 2019-07-17T12:21:02Z

Checkstyle is unhappy, otherwise LGTM

…vals after HADOOP-15679

steveloughran · 2019-07-17T12:32:37Z

LGTM, +1

steveloughran · 2019-07-17T12:54:25Z

Committed to trunk and branch-3.2

JIRA is still open as it is down; if you want to backport to earlier versions use that JIRA, if you want to stop here then just close it

hadoop-yetus · 2019-07-17T14:56:17Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	262	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	1542	trunk passed
+1	compile	1453	trunk passed
+1	checkstyle	57	trunk passed
+1	mvnsite	123	trunk passed
+1	shadedclient	1115	branch has no errors when building and testing our client artifacts.
+1	javadoc	84	trunk passed
0	spotbugs	168	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	160	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	63	the patch passed
+1	compile	1332	the patch passed
+1	javac	1332	the patch passed
-0	checkstyle	59	hadoop-common-project/hadoop-common: The patch generated 1 new + 2 unchanged - 0 fixed = 3 total (was 2)
+1	mvnsite	101	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	883	patch has no errors when building and testing our client artifacts.
+1	javadoc	87	the patch passed
+1	findbugs	180	the patch passed
		_ Other Tests _
+1	unit	636	hadoop-common in the patch passed.
+1	asflicense	54	The patch does not generate ASF License warnings.
		8427

Subsystem	Report/Notes
Docker	Client=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1086/5/artifact/out/Dockerfile
GITHUB PR	#1086
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux ec661f7c13a5 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `19a0018`
Default Java	1.8.0_212
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1086/5/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1086/5/testReport/
Max. process+thread count	1346 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1086/5/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Author: Ray Matharu <rmatharu@linkedin.com> Reviewers: Jagadish<jagadish@apache.org> Closes apache#1086 from rmatharu/test-hotstandbyfix

HADOOP-16341. ShutDownHookManager: Regressed performance on Hook remo…

8416d44

…vals after HADOOP-15679

zeroflag force-pushed the HADOOP-16341-trunk branch from 7cceea0 to 8416d44 Compare July 12, 2019 14:49

HADOOP-16341. ShutDownHookManager: Regressed performance on Hook remo…

be5cf2d

…vals after HADOOP-15679

steveloughran reviewed Jul 15, 2019

View reviewed changes

HADOOP-16341. ShutDownHookManager: Regressed performance on Hook remo…

14ee0e0

…vals after HADOOP-15679

HADOOP-16341. ShutDownHookManager: Regressed performance on Hook remo…

aee28fb

…vals after HADOOP-15679

steveloughran changed the title ~~HADOOP-16341. ShutDownHookManager: Regressed performance on Hook remo…~~ HADOOP-16341. ShutDownHookManager: Regressed performance on Hook removals after HADOOP-15679 Jul 17, 2019

steveloughran closed this Jul 17, 2019

This was referenced Jul 17, 2019

HADOOP-16341. ShutDownHookManager: Regressed performance on Hook removals after HADOOP-15679 #1109

Closed

HADOOP-16341. ShutDownHookManager: Regressed performance on Hook removals after HADOOP-15679 #1110

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-16341. ShutDownHookManager: Regressed performance on Hook removals after HADOOP-15679 #1086

HADOOP-16341. ShutDownHookManager: Regressed performance on Hook removals after HADOOP-15679 #1086

Uh oh!

zeroflag commented Jul 12, 2019 •

edited

Loading

Uh oh!

zeroflag commented Jul 12, 2019

Uh oh!

hadoop-yetus commented Jul 12, 2019

Uh oh!

hadoop-yetus commented Jul 12, 2019

Uh oh!

hadoop-yetus commented Jul 13, 2019

Uh oh!

steveloughran left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented Jul 15, 2019

Uh oh!

zeroflag commented Jul 15, 2019

Uh oh!

zeroflag commented Jul 15, 2019

Uh oh!

steveloughran commented Jul 17, 2019

Uh oh!

steveloughran commented Jul 17, 2019

Uh oh!

steveloughran commented Jul 17, 2019

Uh oh!

hadoop-yetus commented Jul 17, 2019

Uh oh!

Uh oh!

HADOOP-16341. ShutDownHookManager: Regressed performance on Hook removals after HADOOP-15679 #1086

HADOOP-16341. ShutDownHookManager: Regressed performance on Hook removals after HADOOP-15679 #1086

Uh oh!

Conversation

zeroflag commented Jul 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zeroflag commented Jul 12, 2019

Uh oh!

hadoop-yetus commented Jul 12, 2019

Uh oh!

hadoop-yetus commented Jul 12, 2019

Uh oh!

hadoop-yetus commented Jul 13, 2019

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented Jul 15, 2019

Uh oh!

zeroflag commented Jul 15, 2019

Uh oh!

zeroflag commented Jul 15, 2019

Uh oh!

steveloughran commented Jul 17, 2019

Uh oh!

steveloughran commented Jul 17, 2019

Uh oh!

steveloughran commented Jul 17, 2019

Uh oh!

hadoop-yetus commented Jul 17, 2019

Uh oh!

Uh oh!

zeroflag commented Jul 12, 2019 •

edited

Loading