HADOOP-16579. Upgrade to Curator 4.2.0 and ZooKeeper 3.5.5

[symat]

In this PR we upgraded the Apache Curator to 4.0.2 and Apache ZooKeeper to 3.5.5 3.5.6.

While the new Curator and ZooKeeper versions are mostly backward-compatible with the old ones, we encountered a few minor issues still. So far the following changes have been made:

• I added a static initializer for the unit tests using ZooKeeper to enable the four-letter-words diagnostic telnet commands. This is an interface that become disabled by default due to security concerns. (see https://issues.apache.org/jira/browse/ZOOKEEPER-2693) To keep the ZooKeeper 3.4.x behaviour, we enabled it for the tests. Some tests in Hadoop (or other projects using Hadoop-Common) might actually use this feature e.g. to verify the status of ZooKeeper.
• I also fixed ZKFailoverController to look for relevant fail-over ActiveAttempt records. The new ZooKeeper seems to respond quicker during the fail-over tests than the old ZooKeeper, so we made sure to catch all the relevant records by adding a new parameter to ZKFailoverontroller.waitForActiveAttempt().

Update 1:

• On the new curator some of the API calls we use in Hadoop become deprecated, I updated that part in hadoop to not have compile time warnings
• I changed the way how some tests are starting ZooKeeper server (the old server could be started with port 0 and bound to a random open port, but this feature is not working in ZooKeeper 3.5.5 3.5.6, so I choose a random open port before starting the ZooKeeper server in the Hadoop code)

Update 2:

• ZooKeeper 3.5.6 just got released, so I updated the version

[jojochuang]

LGTM

[symat]

ZooKeeper 3.5.6 just got released, I gave it a try. If something goes wrong, I will revert.

I don't think the official announcement has been made, but the new ZooKeeper is already available on Maven Central. It contains some minor fixes and improvements, plus some CVE fixes. See the release note: https://github.com/apache/zookeeper/blob/branch-3.5.6/zookeeper-docs/src/main/resources/markdown/releasenotes.md

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	44	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 3 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	75	Maven dependency ordering for branch
+1	mvninstall	1100	trunk passed
+1	compile	1077	trunk passed
+1	checkstyle	170	trunk passed
+1	mvnsite	319	trunk passed
+1	shadedclient	1283	branch has no errors when building and testing our client artifacts.
+1	javadoc	275	trunk passed
0	spotbugs	114	Used deprecated FindBugs config; considering switching to SpotBugs.
0	findbugs	35	branch/hadoop-project no findbugs output file (findbugsXml.xml)
-1	findbugs	112	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in trunk has 1 extant findbugs warnings.
		_ Patch Compile Tests _
0	mvndep	34	Maven dependency ordering for patch
+1	mvninstall	198	the patch passed
+1	compile	1021	the patch passed
+1	javac	1021	the patch passed
-0	checkstyle	173	root: The patch generated 26 new + 553 unchanged - 6 fixed = 579 total (was 559)
+1	mvnsite	313	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	1	The patch has no ill-formed XML file.
+1	shadedclient	761	patch has no errors when building and testing our client artifacts.
+1	javadoc	287	the patch passed
0	findbugs	32	hadoop-project has no data from findbugs
		_ Other Tests _
+1	unit	30	hadoop-project in the patch passed.
+1	unit	201	hadoop-auth in the patch passed.
+1	unit	575	hadoop-common in the patch passed.
+1	unit	82	hadoop-registry in the patch passed.
+1	unit	168	hadoop-yarn-server-common in the patch passed.
+1	unit	5025	hadoop-yarn-server-resourcemanager in the patch passed.
+1	asflicense	53	The patch does not generate ASF License warnings.
		14072

Subsystem	Report/Notes
Docker	Client=19.03.3 Server=19.03.3 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1656/4/artifact/out/Dockerfile
GITHUB PR	#1656
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
uname	Linux 07449dc4efb8 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 3990ffa
Default Java	1.8.0_222
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-1656/4/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1656/4/artifact/out/diff-checkstyle-root.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1656/4/testReport/
Max. process+thread count	1395 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-common-project/hadoop-registry hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1656/4/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

[symat]

Looks like the build succeeded (the only Findbugs issue was introduced by an independent recent commit of YARN-9773)

[jojochuang]

+1

[steveloughran]

Quick q: I'm seeing TestZKFailoverController failures on hadoop-common test runs on a jenkins server. Does this change it or are the failures there all on our side?

[symat]

do you see this test to fail on the current apache trunk branch?
I also saw it failing after upgrading ZooKeeper in Hadoop, and I had to add a small fix to the ZKFailoverController. After this fix the TestZKFailoverController was passing for me locally and also didn't fail on Yetus.

[steveloughran]

think this is a branch without that patch in; I'll look more closely. thanks