YARN-11702: Fix Yarn over allocating containers #6990

shameersss1 · 2024-08-11T11:24:17Z

Description of PR

Refer https://issues.apache.org/jira/browse/YARN-11702 for more information

How was this patch tested?

Added Unit Test
Manual Testing

For code changes:

[ X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[ NA] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[NA ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ NA] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-yetus · 2024-08-11T16:31:18Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	17m 37s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 41s		Maven dependency ordering for branch
+1 💚	mvninstall	32m 36s		trunk passed
+1 💚	compile	7m 32s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	7m 6s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	1m 56s		trunk passed
+1 💚	mvnsite	2m 15s		trunk passed
+1 💚	javadoc	2m 7s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	1m 55s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	4m 28s		trunk passed
+1 💚	shadedclient	35m 36s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 34s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 27s		the patch passed
+1 💚	compile	6m 54s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	6m 54s		the patch passed
+1 💚	compile	6m 54s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	6m 54s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 53s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt	hadoop-yarn-project/hadoop-yarn: The patch generated 13 new + 362 unchanged - 0 fixed = 375 total (was 362)
+1 💚	mvnsite	2m 7s		the patch passed
+1 💚	javadoc	1m 50s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	1m 46s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	4m 37s		the patch passed
+1 💚	shadedclient	35m 39s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	1m 13s	/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt	hadoop-yarn-api in the patch passed.
+1 💚	unit	109m 25s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	1m 0s		The patch does not generate ASF License warnings.
		305m 46s

Reason	Tests
Failed junit tests	hadoop.yarn.conf.TestYarnConfigurationFields

Subsystem	Report/Notes
Docker	ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6990/1/artifact/out/Dockerfile
GITHUB PR	#6990
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 0f5154611c28 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `f896390`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6990/1/testReport/
Max. process+thread count	955 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6990/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

shameersss1 · 2024-08-12T07:26:44Z

@slfan1989 @p-szucs Could you please review the changes ?

hadoop-yetus · 2024-08-12T12:31:29Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 53s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 14s		Maven dependency ordering for branch
+1 💚	mvninstall	32m 28s		trunk passed
+1 💚	compile	7m 39s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	7m 5s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	2m 1s		trunk passed
+1 💚	mvnsite	3m 16s		trunk passed
+1 💚	javadoc	3m 5s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	3m 0s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	6m 18s		trunk passed
+1 💚	shadedclient	35m 32s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 34s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 4s		the patch passed
+1 💚	compile	7m 0s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	7m 0s		the patch passed
+1 💚	compile	7m 5s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	7m 5s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	1m 52s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt	hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 362 unchanged - 0 fixed = 365 total (was 362)
+1 💚	mvnsite	2m 59s		the patch passed
+1 💚	javadoc	2m 49s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	2m 44s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	6m 35s		the patch passed
+1 💚	shadedclient	35m 46s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 14s		hadoop-yarn-api in the patch passed.
+1 💚	unit	5m 50s		hadoop-yarn-common in the patch passed.
+1 💚	unit	109m 19s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 59s		The patch does not generate ASF License warnings.
		304m 17s

Subsystem	Report/Notes
Docker	ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6990/2/artifact/out/Dockerfile
GITHUB PR	#6990
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux c6f81d04cd2c 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `b75fcdc`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6990/2/testReport/
Max. process+thread count	996 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6990/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-08-16T11:33:38Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 54s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 41s		Maven dependency ordering for branch
+1 💚	mvninstall	32m 46s		trunk passed
+1 💚	compile	7m 32s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	7m 2s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	1m 59s		trunk passed
+1 💚	mvnsite	3m 17s		trunk passed
+1 💚	javadoc	3m 10s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	3m 0s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	6m 12s		trunk passed
+1 💚	shadedclient	37m 15s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 32s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 4s		the patch passed
+1 💚	compile	6m 53s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	6m 53s		the patch passed
+1 💚	compile	6m 57s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	6m 57s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 47s		the patch passed
+1 💚	mvnsite	2m 58s		the patch passed
+1 💚	javadoc	2m 50s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	2m 45s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	6m 37s		the patch passed
+1 💚	shadedclient	35m 34s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 12s		hadoop-yarn-api in the patch passed.
+1 💚	unit	5m 54s		hadoop-yarn-common in the patch passed.
+1 💚	unit	110m 12s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	1m 0s		The patch does not generate ASF License warnings.
		306m 49s

Subsystem	Report/Notes
Docker	ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6990/3/artifact/out/Dockerfile
GITHUB PR	#6990
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 15d358e6946c 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `c1f3596`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6990/3/testReport/
Max. process+thread count	951 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6990/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

shameersss1 · 2024-08-19T09:19:38Z

@slfan1989 @aajisaka Could you please review the changes ?

...main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml

aajisaka · 2024-08-20T07:49:51Z

Hi @slfan1989 would you take a look? To be honest, I'm not familiar with the latest YARN code base and it's possible I might miss something.

hadoop-yetus · 2024-08-22T10:27:28Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	17m 39s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 3s		Maven dependency ordering for branch
+1 💚	mvninstall	32m 22s		trunk passed
+1 💚	compile	8m 1s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	6m 58s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	2m 1s		trunk passed
+1 💚	mvnsite	3m 16s		trunk passed
+1 💚	javadoc	3m 12s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	3m 0s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	6m 21s		trunk passed
+1 💚	shadedclient	35m 25s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 34s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 4s		the patch passed
+1 💚	compile	7m 3s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	7m 3s		the patch passed
+1 💚	compile	6m 58s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	6m 58s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 50s		the patch passed
+1 💚	mvnsite	3m 2s		the patch passed
+1 💚	javadoc	2m 53s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	2m 42s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	6m 37s		the patch passed
+1 💚	shadedclient	35m 49s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 12s		hadoop-yarn-api in the patch passed.
+1 💚	unit	5m 51s		hadoop-yarn-common in the patch passed.
+1 💚	unit	109m 48s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	1m 0s		The patch does not generate ASF License warnings.
		322m 20s

Subsystem	Report/Notes
Docker	ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6990/4/artifact/out/Dockerfile
GITHUB PR	#6990
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 1200633d4eca 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `8da713d`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6990/4/testReport/
Max. process+thread count	938 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6990/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

aajisaka · 2024-09-04T07:12:34Z

Thank you @shameersss1. I'll merge this in this weekend if there's no objection.

slfan1989 · 2024-09-04T10:30:02Z

@aajisaka Sorry I missed some messages. I will review this PR. Please give me 1-2 days.

cc: @shameersss1

zeekling · 2024-09-08T15:36:24Z

Why are multiple requests for Containers sent? This is the key to the problem.

shameersss1 · 2024-09-09T04:58:43Z

Why are multiple requests for Containers sent? This is the key to the problem.

@zeekling ,That's how the AMRMClient is implemented. As long as AM doesn't get the required containers, it keeps on sending the request in every heartbeat untill all the container request is fulfilled.

shameersss1 · 2024-09-09T04:59:54Z

@slfan1989 - Gentle reminder for review.
cc: @aajisaka

slfan1989 · 2024-09-10T03:22:43Z

@shameersss1 Thank you for your contribution. I will complete the review today.

slfan1989 · 2024-09-11T00:10:01Z

...main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java

+    @Override
+    public int hashCode() {
+      final int prime = 31;
+      int result = 1;


We typically use HashCodeBuilder.

slfan1989 · 2024-09-11T00:10:02Z

...main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java

+
+    @Override
+    public boolean equals(Object obj) {
+      if (obj == null) {


EqualsBuilder

slfan1989 · 2024-09-11T00:12:00Z

@shameersss1 Thank you very much for your contribution! From my perspective, it makes sense, and the code quality is high. I have left comments on a few minor details.

zeekling · 2024-09-16T15:00:11Z

The scheduler will not discard resource application requests, so why do we need to apply multiple times?
@shameersss1

shameersss1 · 2024-09-22T09:52:48Z

The scheduler will not discard resource application requests, so why do we need to apply multiple times? @shameersss1

Yes, but the way AMRMClient in Hadoop works in a way that AM always sends pendingconatiner request as part of heartbeat. This is done because of two reasons

AM can dynamically change the request (Eg: Tez have auto parallelism and container reuse)
To make sure AM always went what it wants.

shameersss1 · 2024-09-22T10:23:31Z

@aajisaka @slfan1989 - I have addressed the latest comments - Please review
Thanks

hadoop-yetus · 2024-09-22T15:49:40Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	17m 56s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 43s		Maven dependency ordering for branch
+1 💚	mvninstall	33m 26s		trunk passed
+1 💚	compile	7m 37s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	7m 10s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	2m 0s		trunk passed
+1 💚	mvnsite	3m 17s		trunk passed
+1 💚	javadoc	3m 9s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	3m 0s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	6m 19s		trunk passed
+1 💚	shadedclient	37m 2s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 33s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 4s		the patch passed
+1 💚	compile	6m 56s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	6m 56s		the patch passed
+1 💚	compile	6m 59s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	6m 59s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 51s		the patch passed
+1 💚	mvnsite	2m 58s		the patch passed
+1 💚	javadoc	2m 53s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	2m 46s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	6m 36s		the patch passed
+1 💚	shadedclient	37m 2s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 10s		hadoop-yarn-api in the patch passed.
+1 💚	unit	5m 51s		hadoop-yarn-common in the patch passed.
+1 💚	unit	110m 15s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	1m 0s		The patch does not generate ASF License warnings.
		326m 14s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6990/5/artifact/out/Dockerfile
GITHUB PR	#6990
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 82d88f39fd1a 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `a1f8243`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6990/5/testReport/
Max. process+thread count	956 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6990/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

slfan1989 · 2024-09-23T12:40:19Z

@shameersss1 Thanks for the contribution! If there are no other comments in the next 2 days, we will merge this PR to the trunk branch.

slfan1989 · 2024-09-25T04:14:19Z

@shameersss1 Thanks for the contribution! @aajisaka @zeekling Thanks for the review!

YARN-11702: Fix Yarn over allocating containers

f896390

github-actions bot added YARN trunk labels Aug 11, 2024

Fix checkstyle issue

b75fcdc

Checkstyle fix

c1f3596

aajisaka reviewed Aug 20, 2024

View reviewed changes

...main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java Show resolved Hide resolved

aajisaka reviewed Aug 20, 2024

View reviewed changes

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml Outdated Show resolved Hide resolved

Add verbose to the config description

8da713d

shameersss1 requested a review from aajisaka August 26, 2024 06:37

aajisaka approved these changes Sep 4, 2024

View reviewed changes

slfan1989 approved these changes Sep 11, 2024

View reviewed changes

slfan1989 reviewed Sep 11, 2024

View reviewed changes

Use EqualsBuilder and HashBuilder

a1f8243

slfan1989 approved these changes Sep 23, 2024

View reviewed changes

slfan1989 merged commit 21ec686 into apache:trunk Sep 25, 2024
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

YARN-11702: Fix Yarn over allocating containers #6990

YARN-11702: Fix Yarn over allocating containers #6990

shameersss1 commented Aug 11, 2024

hadoop-yetus commented Aug 11, 2024

shameersss1 commented Aug 12, 2024 •

edited

Loading

hadoop-yetus commented Aug 12, 2024

hadoop-yetus commented Aug 16, 2024

shameersss1 commented Aug 19, 2024

aajisaka commented Aug 20, 2024

hadoop-yetus commented Aug 22, 2024

aajisaka commented Sep 4, 2024

slfan1989 commented Sep 4, 2024

zeekling commented Sep 8, 2024

shameersss1 commented Sep 9, 2024 •

edited

Loading

shameersss1 commented Sep 9, 2024

slfan1989 commented Sep 10, 2024

slfan1989 Sep 11, 2024

shameersss1 Sep 22, 2024

slfan1989 Sep 11, 2024

shameersss1 Sep 22, 2024

slfan1989 commented Sep 11, 2024

zeekling commented Sep 16, 2024 •

edited

Loading

shameersss1 commented Sep 22, 2024

shameersss1 commented Sep 22, 2024

hadoop-yetus commented Sep 22, 2024

slfan1989 commented Sep 23, 2024

slfan1989 commented Sep 25, 2024

YARN-11702: Fix Yarn over allocating containers #6990

YARN-11702: Fix Yarn over allocating containers #6990

Conversation

shameersss1 commented Aug 11, 2024

Description of PR

How was this patch tested?

For code changes:

hadoop-yetus commented Aug 11, 2024

shameersss1 commented Aug 12, 2024 • edited Loading

hadoop-yetus commented Aug 12, 2024

hadoop-yetus commented Aug 16, 2024

shameersss1 commented Aug 19, 2024

aajisaka commented Aug 20, 2024

hadoop-yetus commented Aug 22, 2024

aajisaka commented Sep 4, 2024

slfan1989 commented Sep 4, 2024

zeekling commented Sep 8, 2024

shameersss1 commented Sep 9, 2024 • edited Loading

shameersss1 commented Sep 9, 2024

slfan1989 commented Sep 10, 2024

slfan1989 Sep 11, 2024

Choose a reason for hiding this comment

shameersss1 Sep 22, 2024

Choose a reason for hiding this comment

slfan1989 Sep 11, 2024

Choose a reason for hiding this comment

shameersss1 Sep 22, 2024

Choose a reason for hiding this comment

slfan1989 commented Sep 11, 2024

zeekling commented Sep 16, 2024 • edited Loading

shameersss1 commented Sep 22, 2024

shameersss1 commented Sep 22, 2024

hadoop-yetus commented Sep 22, 2024

slfan1989 commented Sep 23, 2024

slfan1989 commented Sep 25, 2024

shameersss1 commented Aug 12, 2024 •

edited

Loading

shameersss1 commented Sep 9, 2024 •

edited

Loading

zeekling commented Sep 16, 2024 •

edited

Loading