YARN-11082 AbstractCSQueue#canAssignToThisQueue DRF should use node partition reource as denominator

[BrightK7]

Description of PR

AbstractCSQueue#canAssignToThisQueue will check current queue useage and limit, and DRF will use cluster resource as denominator to check which resource is dominated and comapre the ratio however if our cluster's nodes resource are not blance such as there is larger proportion of memory/vores, then DRF will chose wrong dominated resource.
For Example our cluster's total resouce are <memory:175117312, vCores:40222> the ratio is 1 vores : 4.25 GB, and the ratio changed to 1 : 4.8 under node label x.

2021-12-09 10:24:37,069 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: assignedContainer application attempt=appattempt_xxx_xxx container=xxx queue=root.a.a1.a2 clusterResource=<memory:175117312, vCores:40222> type=RACK_LOCAL requestedPartition=x
2021-12-09 10:24:37,069 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue: Used resource=<memory:3381248, vCores:687> exceeded maxResourceLimit of the queue =<memory:3420315, vCores:687>

2021-12-09 10:24:37,069 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Failed to accept allocation proposal

clusterResource = <memory:175117312, vCores:40222>
usedExceptKillable = <memory:3381248, vCores:687>
currentLimitResource = <memory:3420315, vCores:687>

currentLimitResource:
memory : 3381248/175117312 = 0.01930847362
vCores : 687/40222 = 0.01708020486

usedExceptKillable:
memory : 3384320/175117312 = 0.01932601615
vCores : 688/40222 = 0.01710506687

DRF will think memory is dominated resource and compare the ratio of memeory in this scenario

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 53s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	34m 59s		trunk passed
+1 💚	compile	1m 6s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	compile	0m 55s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 45s		trunk passed
+1 💚	mvnsite	1m 1s		trunk passed
+1 💚	javadoc	0m 49s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 43s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 59s		trunk passed
+1 💚	shadedclient	24m 27s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 54s		the patch passed
+1 💚	compile	1m 0s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javac	1m 0s		the patch passed
+1 💚	compile	0m 49s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 49s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 38s		the patch passed
+1 💚	mvnsite	0m 54s		the patch passed
+1 💚	javadoc	0m 40s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 37s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	2m 0s		the patch passed
+1 💚	shadedclient	24m 12s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	100m 34s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 29s		The patch does not generate ASF License warnings.
		199m 20s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4043/1/artifact/out/Dockerfile
GITHUB PR	#4043
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 61adbef8f234 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 5c25f3b
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4043/1/testReport/
Max. process+thread count	917 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4043/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.