YARN-10302. Make FairScheduler node comparator configurable

[billonahill]

Add yarn.scheduler.fair.node.comparator.class config to control the FairScheduler node comparator.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 31s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1 💚	mvninstall	25m 49s	trunk passed
+1 💚	compile	1m 4s	trunk passed
+1 💚	checkstyle	0m 45s	trunk passed
+1 💚	mvnsite	1m 5s	trunk passed
+1 💚	shadedclient	18m 1s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 31s	trunk passed
+0 🆗	spotbugs	1m 44s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 40s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 45s	the patch passed
+1 💚	compile	0m 40s	the patch passed
+1 💚	javac	0m 40s	the patch passed
-0 ⚠️	checkstyle	0m 29s	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 10 new + 33 unchanged - 0 fixed = 43 total (was 33)
+1 💚	mvnsite	0m 44s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	15m 42s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 28s	the patch passed
-1 ❌	findbugs	1m 49s	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
		_ Other Tests _
+1 💚	unit	93m 23s	hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 27s	The patch does not generate ASF License warnings.
		165m 37s

Reason	Tests
FindBugs	module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
	org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$NodeAvailableResourceComparator implements Comparator but not Serializable At FairScheduler.java:Serializable At FairScheduler.java:[lines 1080-1087]

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2044/1/artifact/out/Dockerfile
GITHUB PR	#2044
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux f4eb0fcea521 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 9fe4c37
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-2044/1/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-2044/1/artifact/out/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2044/1/testReport/
Max. process+thread count	834 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2044/1/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	22m 45s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1 💚	mvninstall	19m 26s	trunk passed
+1 💚	compile	0m 47s	trunk passed
+1 💚	checkstyle	0m 39s	trunk passed
+1 💚	mvnsite	0m 52s	trunk passed
+1 💚	shadedclient	15m 19s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 35s	trunk passed
+0 🆗	spotbugs	1m 42s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 39s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 49s	the patch passed
+1 💚	compile	0m 41s	the patch passed
+1 💚	javac	0m 41s	the patch passed
+1 💚	checkstyle	0m 31s	the patch passed
+1 💚	mvnsite	0m 46s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	16m 7s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 32s	the patch passed
+1 💚	findbugs	1m 56s	the patch passed
		_ Other Tests _
-1 ❌	unit	55m 5s	hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 34s	The patch does not generate ASF License warnings.
		139m 59s

Reason	Tests
Failed junit tests	hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
	hadoop.yarn.server.resourcemanager.scheduler.policy.TestFairOrderingPolicy
	hadoop.yarn.server.resourcemanager.TestRMAdminService
	hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerHealth
	hadoop.yarn.server.resourcemanager.TestAppManager

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2044/2/artifact/out/Dockerfile
GITHUB PR	#2044
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux b4953fc558fe 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 6288e15
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-2044/2/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2044/2/testReport/
Max. process+thread count	810 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2044/2/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[billonahill]

Regarding the test4tests failure, this patch preserves backward compatibility and effects lines of code already tested, hence no new tests are added.

It seems like all the failing unit tests are getting OOM errors.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 18s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1 💚	mvninstall	20m 21s	trunk passed
+1 💚	compile	0m 58s	trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	compile	0m 48s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	checkstyle	0m 39s	trunk passed
+1 💚	mvnsite	0m 57s	trunk passed
+1 💚	shadedclient	15m 33s	branch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 26s	hadoop-yarn-server-resourcemanager in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.
+1 💚	javadoc	0m 33s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+0 🆗	spotbugs	1m 54s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 52s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 50s	the patch passed
+1 💚	compile	0m 57s	the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
-1 ❌	javac	0m 57s	hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 1 new + 42 unchanged - 1 fixed = 43 total (was 43)
+1 💚	compile	0m 48s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	javac	0m 48s	the patch passed
+1 💚	checkstyle	0m 37s	the patch passed
+1 💚	mvnsite	0m 47s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	14m 9s	patch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 24s	hadoop-yarn-server-resourcemanager in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.
+1 💚	javadoc	0m 31s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	findbugs	1m 52s	the patch passed
		_ Other Tests _
+1 💚	unit	110m 9s	hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 31s	The patch does not generate ASF License warnings.
		176m 16s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2044/3/artifact/out/Dockerfile
GITHUB PR	#2044
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 21bfe4df1734 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 05b3337
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09
javadoc	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2044/3/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt
javac	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2044/3/artifact/out/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt
javadoc	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2044/3/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2044/3/testReport/
Max. process+thread count	870 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2044/3/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.