HDFS-14755. [Dynamometer] Hadoop-2 DataNode fail to start

[tasanuma]

No description provided.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	154	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	1614	trunk passed
+1	compile	29	trunk passed
+1	checkstyle	26	trunk passed
+1	mvnsite	31	trunk passed
+1	shadedclient	925	branch has no errors when building and testing our client artifacts.
+1	javadoc	30	trunk passed
0	spotbugs	64	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	60	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	31	the patch passed
+1	compile	19	the patch passed
+1	javac	19	the patch passed
+1	checkstyle	14	the patch passed
+1	mvnsite	25	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	970	patch has no errors when building and testing our client artifacts.
+1	javadoc	25	the patch passed
+1	findbugs	49	the patch passed
		_ Other Tests _
+1	unit	482	hadoop-dynamometer-infra in the patch passed.
+1	asflicense	43	The patch does not generate ASF License warnings.
		4674

Subsystem	Report/Notes
Docker	Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1320/1/artifact/out/Dockerfile
GITHUB PR	#1320
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 7f6de639b055 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 6244502
Default Java	1.8.0_212
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1320/1/testReport/
Max. process+thread count	927 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra U: hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1320/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

[xkrogen]

@tasanuma, thanks for submitting this! I don't think this will work if you have multiple storage locations configured, which is a common pattern. Is there anything we can do which will both support multiple storage locations, but also be backwards-compatible?

[tasanuma]

@xkrogen Thanks for your review!
In fact, I tested multiple storage locations and seems it works fine.

[xkrogen]

Can you explain why it works? This is the code MiniDFSCluster uses to parse it (within GenericTestUtils):

  public static File getTestDir() {
    String prop = System.getProperty(SYSPROP_TEST_DATA_DIR, DEFAULT_TEST_DATA_DIR);
    if (prop.isEmpty()) {
      // corner case: property is there but empty
      prop = DEFAULT_TEST_DATA_DIR;
    }
    File dir = new File(prop).getAbsoluteFile();

(SYSPROP_TEST_DATA_DIR is the same as PROP_TEST_BUILD_DATA)

I would expect new File(prop) to complain if you give it a comma-separated list of directories.

[tasanuma]

Thanks for your explanation, @xkrogen.

I see. I'm not sure why my job worked, but it should be one directory there. So if we have multiple storage locations, we should use the first of them here, right?

Updated the PR. It handles that case while keeping the compatibility between hadoop-2 and hadoop-3.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	144	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	1542	trunk passed
+1	compile	26	trunk passed
+1	checkstyle	26	trunk passed
+1	mvnsite	28	trunk passed
+1	shadedclient	904	branch has no errors when building and testing our client artifacts.
+1	javadoc	29	trunk passed
0	spotbugs	56	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	50	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	31	the patch passed
+1	compile	22	the patch passed
+1	javac	22	the patch passed
+1	checkstyle	15	the patch passed
+1	mvnsite	26	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	977	patch has no errors when building and testing our client artifacts.
+1	javadoc	21	the patch passed
+1	findbugs	62	the patch passed
		_ Other Tests _
+1	unit	489	hadoop-dynamometer-infra in the patch passed.
+1	asflicense	47	The patch does not generate ASF License warnings.
		4574

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1320/2/artifact/out/Dockerfile
GITHUB PR	#1320
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux b3eae61d6407 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / 5e156b9
Default Java	1.8.0_222
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1320/2/testReport/
Max. process+thread count	925 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra U: hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1320/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	102	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
-1	test4tests	0	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1	mvninstall	1473	trunk passed
+1	compile	28	trunk passed
+1	checkstyle	25	trunk passed
+1	mvnsite	31	trunk passed
+1	shadedclient	881	branch has no errors when building and testing our client artifacts.
+1	javadoc	27	trunk passed
0	spotbugs	52	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	47	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	32	the patch passed
+1	compile	20	the patch passed
+1	javac	20	the patch passed
+1	checkstyle	17	the patch passed
+1	mvnsite	24	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	946	patch has no errors when building and testing our client artifacts.
+1	javadoc	21	the patch passed
+1	findbugs	50	the patch passed
		_ Other Tests _
+1	unit	502	hadoop-dynamometer-infra in the patch passed.
+1	asflicense	38	The patch does not generate ASF License warnings.
		4401

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1320/3/artifact/out/Dockerfile
GITHUB PR	#1320
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux eea15bb01f2a 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / ee7c261
Default Java	1.8.0_212
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1320/3/testReport/
Max. process+thread count	927 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra U: hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1320/3/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

[xkrogen]

I see. The problem is just that getUri() is private in branch-2, so we can still leverage the functionality of DataNode.getStorageLocations, we just can't directly convert it to a URI. This will fail in some scenarios, e.g. a PROVIDED storage type, but I think this is fine. +1 from me.

[xkrogen]

+1

[tasanuma]

Thanks!