YARN-11413. Fix Junit Test ERROR Introduced By YARN-6412. #5289

slfan1989 · 2023-01-10T15:43:33Z

JIRA: YARN-11413. Fix Junit Test ERROR Introduced By YARN-6412.

YARN-6412 (#5242) caused TestCommonConfigurationFields#testCompareXmlAgainstConfigurationClass to be wrong.

We can see the following error message
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5193/15/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt

[ERROR] testCompareXmlAgainstConfigurationClass  Time elapsed: 0.292 s  <<< FAILURE!
java.lang.AssertionError: yarn-default.xml has 2 properties missing in  class org.apache.hadoop.yarn.conf.YarnConfiguration Entries:   yarn.nodemanager.aux-services.%s.classpath  yarn.nodemanager.aux-services.%s.system-classes expected:<0> but was:<2>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.apache.hadoop.conf.TestConfigurationFieldsBase.testCompareXmlAgainstConfigurationClass(TestConfigurationFieldsBase.java:540)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	.....

slfan1989 · 2023-01-10T15:49:33Z

@brumi1024 @riyakhdl I found that YARN-6412 (#5242) introduces a unit test bug, I think it is necessary to add instructions in yarn-site.xml, but we should let testCompareXmlAgainstConfigurationClass ignore the newly added property yarn.nodemanager.aux-services.%s.classpath 、yarn.nodemanager.aux-services.%s.system-classes.

hadoop-yetus · 2023-01-10T19:26:40Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 1s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	45m 51s		trunk passed
+1 💚	compile	23m 7s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	compile	20m 33s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	checkstyle	1m 14s		trunk passed
+1 💚	mvnsite	1m 42s		trunk passed
-1 ❌	javadoc	1m 13s	/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-common in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	0m 49s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	2m 40s		trunk passed
+1 💚	shadedclient	25m 36s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 4s		the patch passed
+1 💚	compile	22m 30s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javac	22m 30s		the patch passed
+1 💚	compile	20m 38s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	javac	20m 38s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 5s		the patch passed
+1 💚	mvnsite	1m 40s		the patch passed
-1 ❌	javadoc	1m 6s	/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-common in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	0m 51s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	2m 41s		the patch passed
+1 💚	shadedclient	25m 5s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 26s		hadoop-common in the patch passed.
+1 💚	asflicense	1m 4s		The patch does not generate ASF License warnings.
		220m 10s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5289/1/artifact/out/Dockerfile
GITHUB PR	#5289
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux b5eda3c7f44c 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `c014609`
Default Java	Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5289/1/testReport/
Max. process+thread count	1278 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5289/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

brumi1024

Thanks @slfan1989 for the quick reaction! I agree that the test is at fault here, because the properties that doesn't match this pattern are filtered out from the Configuration file, so basically every configuration property that has the %s in its name for string replacement will break this test once it gets documented, which is exactly the opposite of what this test should be doing. However instead of skipping these props, can you please update the pattern?

slfan1989 · 2023-01-11T09:55:46Z

Thanks @slfan1989 for the quick reaction! I agree that the test is at fault here, because the properties that doesn't match this pattern are filtered out from the Configuration file, so basically every configuration property that has the %s in its name for string replacement will break this test once it gets documented, which is exactly the opposite of what this test should be doing. However instead of skipping these props, can you please update the pattern?

@brumi1024 Thanks for your suggestion, I will fix it as soon as possible.

goiri · 2023-01-11T18:27:55Z

@brumi1024 can you take a look?

brumi1024 · 2023-01-11T20:01:18Z

@goiri see my comment above. I think adding these two props to the skipped list is just hiding a different problem, the pattern should be fixed. I can provide an update tomorrow if @slfan1989 doesn’t have the time for it.

slfan1989 · 2023-01-11T22:41:30Z

@goiri @brumi1024 Thank you very much for helping to review the code, I'll try to fix it.

slfan1989 · 2023-01-12T02:58:10Z

@goiri @brumi1024

Code modification instructions are as follows:

1.Modify the pattern and add % to identify attributes containing %s.
yarn.nodemanager.aux-services.%s.classpath and yarn.nodemanager.aux-services.%s.system-classes can be identified.

2.There are some properties in YarnConfiguration containing %s, but these properties do not exist in yarn-site.xml, this part of properties, I will add them to configurationPropsToSkipCompare.

The attribute information is as follows:

yarn.log-aggregation.%s.remote-app-log-dir
yarn.log-aggregation.%s.remote-app-log-dir-suffix
yarn.log-aggregation.file-controller.%s.class
yarn.nodemanager.aux-services.%s.class
yarn.nodemanager.health-checker.%s.interval-ms
yarn.nodemanager.health-checker.%s.opts
yarn.nodemanager.health-checker.%s.path
yarn.nodemanager.health-checker.%s.timeout-ms
yarn.nodemanager.aux-services.%s.remote-classpath
yarn.nodemanager.runtime.linux.%s.class

hadoop-yetus · 2023-01-12T06:17:23Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 40s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 38s		Maven dependency ordering for branch
+1 💚	mvninstall	31m 3s		trunk passed
+1 💚	compile	23m 4s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	compile	20m 23s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	checkstyle	3m 48s		trunk passed
+1 💚	mvnsite	2m 43s		trunk passed
-1 ❌	javadoc	1m 15s	/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-common in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	1m 42s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	4m 50s		trunk passed
+1 💚	shadedclient	24m 46s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 29s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 51s		the patch passed
+1 💚	compile	22m 19s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javac	22m 19s		the patch passed
+1 💚	compile	20m 26s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	javac	20m 26s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	3m 37s	/results-checkstyle-root.txt	root: The patch generated 2 new + 49 unchanged - 0 fixed = 51 total (was 49)
+1 💚	mvnsite	2m 45s		the patch passed
-1 ❌	javadoc	1m 5s	/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-common in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	1m 39s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	4m 57s		the patch passed
+1 💚	shadedclient	24m 58s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 20s		hadoop-common in the patch passed.
-1 ❌	unit	1m 16s	/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt	hadoop-yarn-api in the patch passed.
+1 💚	asflicense	1m 2s		The patch does not generate ASF License warnings.
		238m 43s

Reason	Tests
Failed junit tests	hadoop.yarn.conf.TestYarnConfigurationFields

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5289/2/artifact/out/Dockerfile
GITHUB PR	#5289
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 1569700e1dda 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `f240bd3`
Default Java	Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5289/2/testReport/
Max. process+thread count	1320 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5289/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

brumi1024

Thanks @slfan1989. The latest patch LGTM, and the failing test run successfully locally. Let's wait for a build.

hadoop-yetus · 2023-01-12T12:21:03Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 59s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 22s		Maven dependency ordering for branch
+1 💚	mvninstall	31m 24s		trunk passed
+1 💚	compile	23m 15s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	compile	20m 25s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	checkstyle	3m 47s		trunk passed
+1 💚	mvnsite	2m 43s		trunk passed
-1 ❌	javadoc	1m 15s	/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-common in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	1m 41s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	4m 46s		trunk passed
+1 💚	shadedclient	24m 52s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 29s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 49s		the patch passed
+1 💚	compile	22m 28s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javac	22m 28s		the patch passed
+1 💚	compile	20m 47s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	javac	20m 47s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	3m 34s		the patch passed
+1 💚	mvnsite	2m 45s		the patch passed
-1 ❌	javadoc	1m 7s	/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-common in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	1m 41s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	4m 58s		the patch passed
+1 💚	shadedclient	25m 3s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 25s		hadoop-common in the patch passed.
+1 💚	unit	1m 16s		hadoop-yarn-api in the patch passed.
+1 💚	asflicense	1m 2s		The patch does not generate ASF License warnings.
		239m 47s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5289/3/artifact/out/Dockerfile
GITHUB PR	#5289
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux e9d03bba0c2e 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ae84b75`
Default Java	Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5289/3/testReport/
Max. process+thread count	1693 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5289/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

slfan1989 · 2023-01-12T13:23:59Z

@brumi1024 Can you help to review this PR again? Thank you very much!

brumi1024 · 2023-01-12T13:29:18Z

@slfan1989 merged to trunk.

slfan1989 · 2023-01-12T13:35:11Z

@brumi1024 Thank you very much for helping to review the code!

YARN-11413. Fix Junit Test ERROR Introduced By YARN-6412.

c014609

slfan1989 mentioned this pull request Jan 10, 2023

YARN-11374. [Federation] Support refreshSuperUserGroupsConfiguration、refreshUserToGroupsMappings API's for Federation. #5193

Merged

brumi1024 requested changes Jan 11, 2023

View reviewed changes

goiri approved these changes Jan 11, 2023

View reviewed changes

YARN-11413. Fix CheckStyle.

f240bd3

YARN-11413. Fix CheckStyle.

ae84b75

brumi1024 approved these changes Jan 12, 2023

View reviewed changes

brumi1024 merged commit 3d21cff into apache:trunk Jan 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

YARN-11413. Fix Junit Test ERROR Introduced By YARN-6412. #5289

YARN-11413. Fix Junit Test ERROR Introduced By YARN-6412. #5289

Uh oh!

slfan1989 commented Jan 10, 2023 •

edited

Loading

Uh oh!

slfan1989 commented Jan 10, 2023

Uh oh!

hadoop-yetus commented Jan 10, 2023

Uh oh!

brumi1024 left a comment

Uh oh!

slfan1989 commented Jan 11, 2023

Uh oh!

goiri commented Jan 11, 2023

Uh oh!

brumi1024 commented Jan 11, 2023

Uh oh!

slfan1989 commented Jan 11, 2023

Uh oh!

slfan1989 commented Jan 12, 2023

Uh oh!

hadoop-yetus commented Jan 12, 2023

Uh oh!

brumi1024 left a comment

Uh oh!

hadoop-yetus commented Jan 12, 2023

Uh oh!

slfan1989 commented Jan 12, 2023

Uh oh!

brumi1024 commented Jan 12, 2023

Uh oh!

slfan1989 commented Jan 12, 2023

Uh oh!

Uh oh!

YARN-11413. Fix Junit Test ERROR Introduced By YARN-6412. #5289

YARN-11413. Fix Junit Test ERROR Introduced By YARN-6412. #5289

Uh oh!

Conversation

slfan1989 commented Jan 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

slfan1989 commented Jan 10, 2023

Uh oh!

hadoop-yetus commented Jan 10, 2023

Uh oh!

brumi1024 left a comment

Choose a reason for hiding this comment

Uh oh!

slfan1989 commented Jan 11, 2023

Uh oh!

goiri commented Jan 11, 2023

Uh oh!

brumi1024 commented Jan 11, 2023

Uh oh!

slfan1989 commented Jan 11, 2023

Uh oh!

slfan1989 commented Jan 12, 2023

Uh oh!

hadoop-yetus commented Jan 12, 2023

Uh oh!

brumi1024 left a comment

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Jan 12, 2023

Uh oh!

slfan1989 commented Jan 12, 2023

Uh oh!

brumi1024 commented Jan 12, 2023

Uh oh!

slfan1989 commented Jan 12, 2023

Uh oh!

Uh oh!

slfan1989 commented Jan 10, 2023 •

edited

Loading