HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. #2058

umamaheswararao · 2020-06-06T02:30:05Z

https://issues.apache.org/jira/browse/HDFS-15394

…e>.impl classes in core-default.xml bydefault.

hadoop-yetus · 2020-06-06T04:12:58Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 32s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1 💚	mvninstall	19m 13s	trunk passed
+1 💚	compile	17m 8s	trunk passed
+1 💚	mvnsite	1m 25s	trunk passed
+1 💚	shadedclient	52m 13s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 4s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 51s	the patch passed
+1 💚	compile	16m 19s	the patch passed
+1 💚	javac	16m 19s	the patch passed
+1 💚	mvnsite	1m 26s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 1s	The patch has no ill-formed XML file.
+1 💚	shadedclient	14m 19s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 4s	the patch passed
		_ Other Tests _
-1 ❌	unit	9m 57s	hadoop-common in the patch passed.
+1 💚	asflicense	0m 51s	The patch does not generate ASF License warnings.
		101m 48s

Reason	Tests
Failed junit tests	hadoop.conf.TestCommonConfigurationFields

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2058/1/artifact/out/Dockerfile
GITHUB PR	#2058
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml
uname	Linux c0e8959df699 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `e7dd027`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-2058/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2058/1/testReport/
Max. process+thread count	2205 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2058/1/console
versions	git=2.17.1 maven=3.6.0
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2020-06-06T04:38:37Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 11s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1 💚	mvninstall	22m 9s	trunk passed
+1 💚	compile	19m 21s	trunk passed
+1 💚	mvnsite	1m 32s	trunk passed
+1 💚	shadedclient	61m 26s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 59s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	1m 1s	the patch passed
+1 💚	compile	21m 32s	the patch passed
+1 💚	javac	21m 32s	the patch passed
+1 💚	mvnsite	1m 45s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 3s	The patch has no ill-formed XML file.
+1 💚	shadedclient	18m 52s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 58s	the patch passed
		_ Other Tests _
-1 ❌	unit	10m 38s	hadoop-common in the patch passed.
+1 💚	asflicense	0m 53s	The patch does not generate ASF License warnings.
		122m 24s

Reason	Tests
Failed junit tests	hadoop.conf.TestCommonConfigurationFields

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2058/2/artifact/out/Dockerfile
GITHUB PR	#2058
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml
uname	Linux 9dee42a8ec14 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `e7dd027`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-2058/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2058/2/testReport/
Max. process+thread count	1593 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2058/2/console
versions	git=2.17.1 maven=3.6.0
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ayushtkn · 2020-06-06T05:17:58Z

Thanx @umamaheswararao for the work here.

The TestCommonConfigurationFields is related, It expects the configurations to be added in CommonConfigurationKeys.java as well
Apart from this, The HDFS tests didn't run and I feel couple of them, testing invalid scheme may fail now like these are failing now :

TestViewFileSystemOverloadSchemeWithHdfsScheme#testInvalidOverloadSchemeTargetFS
TestViewFSOverloadSchemeWithMountTableConfigInHDFS#testInvalidOverloadSchemeTargetFS

There are couple of places, where these configurations were explicitly defined, mostly I guess in tests, You may remove couple of them now, which ever places you remember

umamaheswararao · 2020-06-06T05:32:58Z

I see this test is asserting xml vs config class. However, the idea is not to expose this configurations to users. I am building the key dynamically in using
final String fsImplConf = String.format(
FsConstants.FS_VIEWFS_OVERLOAD_SCHEME_TARGET_FS_IMPL_PATTERN,
uri.getScheme());
They are not necessarily added in config class as they will not be useful.
Let me think what other way we can handled with out exposing in config classes.

umamaheswararao · 2020-06-06T06:42:23Z

I have addressed test failures. Please take a look. Thanks
As we don't use that keys directly, but they will be constructed dynamically and used in source code. We don't want to add keys just for tests only. So, I added them in xmlPropsToSkipCompare

...roject/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java

umamaheswararao · 2020-06-06T07:59:00Z

Alternatively I am also thinking ServiceLoader based impl class loading automatically which implements FileSystem, instead of depending on config parameters. That needs more testing from my side. If that works,I will file another JIRA and get rid of this config items. But for now, this configs should work.

hadoop-yetus · 2020-06-06T11:12:53Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 13s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
+0 🆗	mvndep	0m 21s	Maven dependency ordering for branch
+1 💚	mvninstall	23m 8s	trunk passed
+1 💚	compile	21m 51s	trunk passed
+1 💚	checkstyle	3m 22s	trunk passed
+1 💚	mvnsite	3m 13s	trunk passed
+1 💚	shadedclient	22m 29s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 41s	trunk passed
+0 🆗	spotbugs	3m 32s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	5m 42s	trunk passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 23s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 20s	the patch passed
+1 💚	compile	20m 3s	the patch passed
+1 💚	javac	20m 3s	the patch passed
+1 💚	checkstyle	3m 8s	the patch passed
+1 💚	mvnsite	2m 51s	the patch passed
+1 💚	whitespace	0m 1s	The patch has no whitespace issues.
+1 💚	xml	0m 1s	The patch has no ill-formed XML file.
+1 💚	shadedclient	15m 42s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 42s	the patch passed
+1 💚	findbugs	6m 1s	the patch passed
		_ Other Tests _
-1 ❌	unit	10m 30s	hadoop-common in the patch passed.
-1 ❌	unit	132m 2s	hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 24s	The patch does not generate ASF License warnings.
		276m 13s

Reason	Tests
Failed junit tests	hadoop.security.TestFixKerberosTicketOrder
	hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy
	hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics
	hadoop.hdfs.TestReconstructStripedFile
	hadoop.hdfs.TestGetFileChecksum
	hadoop.hdfs.TestSafeModeWithStripedFileWithRandomECPolicy
	hadoop.hdfs.TestRollingUpgrade
	hadoop.fs.viewfs.TestViewFileSystemOverloadSchemeHdfsFileSystemContract

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2058/3/artifact/out/Dockerfile
GITHUB PR	#2058
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle
uname	Linux 7039928ac464 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `cc671b1`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-2058/3/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-2058/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2058/3/testReport/
Max. process+thread count	2714 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2058/3/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2020-06-06T11:18:20Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 37s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 2 new or modified test files.
		_ trunk Compile Tests _
+0 🆗	mvndep	0m 26s	Maven dependency ordering for branch
+1 💚	mvninstall	23m 0s	trunk passed
+1 💚	compile	21m 39s	trunk passed
+1 💚	checkstyle	2m 57s	trunk passed
+1 💚	mvnsite	2m 50s	trunk passed
+1 💚	shadedclient	21m 32s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 54s	trunk passed
+0 🆗	spotbugs	3m 35s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	5m 48s	trunk passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 24s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 23s	the patch passed
+1 💚	compile	19m 37s	the patch passed
+1 💚	javac	19m 37s	the patch passed
+1 💚	checkstyle	2m 54s	the patch passed
+1 💚	mvnsite	2m 54s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 2s	The patch has no ill-formed XML file.
+1 💚	shadedclient	15m 35s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	1m 51s	the patch passed
+1 💚	findbugs	6m 27s	the patch passed
		_ Other Tests _
-1 ❌	unit	10m 29s	hadoop-common in the patch passed.
-1 ❌	unit	133m 45s	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 54s	The patch does not generate ASF License warnings.
		277m 0s

Reason	Tests
Failed junit tests	hadoop.security.TestRaceWhenRelogin
	hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy
	hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
	hadoop.hdfs.TestReconstructStripedFile
	hadoop.hdfs.TestGetFileChecksum
	hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
	hadoop.fs.viewfs.TestViewFileSystemOverloadSchemeHdfsFileSystemContract

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2058/4/artifact/out/Dockerfile
GITHUB PR	#2058
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle
uname	Linux 552dbaa355fd 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `cc671b1`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-2058/4/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-2058/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2058/4/testReport/
Max. process+thread count	2985 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2058/4/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ayushtkn · 2020-06-06T13:48:37Z

Sound great @umamaheswararao
The test failures aren't related.
TestViewFileSystemOverloadSchemeHdfsFileSystemContract looked suspicious but isn't related to this change. Raised HDFS-15396 for that.

+1

umamaheswararao · 2020-06-06T15:11:12Z

Thanks a lot, @ayushtkn for the review!

umamaheswararao added 2 commits June 5, 2020 19:28

HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<schem…

473e89a

…e>.impl classes in core-default.xml bydefault.

HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<schem…

f373657

…e>.impl classes in core-default.xml bydefault.

umamaheswararao added 2 commits June 5, 2020 23:35

Fixed test failures.

1f27f7d

Fixed test failures.

eaa403c

ayushtkn reviewed Jun 6, 2020

View reviewed changes

...roject/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java Show resolved Hide resolved

ayushtkn approved these changes Jun 6, 2020

View reviewed changes

umamaheswararao merged commit 3ca1529 into apache:trunk Jun 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. #2058

HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. #2058

Uh oh!

umamaheswararao commented Jun 6, 2020

Uh oh!

hadoop-yetus commented Jun 6, 2020

Uh oh!

hadoop-yetus commented Jun 6, 2020

Uh oh!

ayushtkn commented Jun 6, 2020

Uh oh!

umamaheswararao commented Jun 6, 2020

Uh oh!

umamaheswararao commented Jun 6, 2020

Uh oh!

Uh oh!

umamaheswararao commented Jun 6, 2020

Uh oh!

hadoop-yetus commented Jun 6, 2020

Uh oh!

hadoop-yetus commented Jun 6, 2020

Uh oh!

ayushtkn commented Jun 6, 2020

Uh oh!

umamaheswararao commented Jun 6, 2020

Uh oh!

Uh oh!

HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. #2058

HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. #2058

Uh oh!

Conversation

umamaheswararao commented Jun 6, 2020

Uh oh!

hadoop-yetus commented Jun 6, 2020

Uh oh!

hadoop-yetus commented Jun 6, 2020

Uh oh!

ayushtkn commented Jun 6, 2020

Uh oh!

umamaheswararao commented Jun 6, 2020

Uh oh!

umamaheswararao commented Jun 6, 2020

Uh oh!

Uh oh!

umamaheswararao commented Jun 6, 2020

Uh oh!

hadoop-yetus commented Jun 6, 2020

Uh oh!

hadoop-yetus commented Jun 6, 2020

Uh oh!

ayushtkn commented Jun 6, 2020

Uh oh!

umamaheswararao commented Jun 6, 2020

Uh oh!

Uh oh!