HDFS-15416. DataStorage#addStorageLocations() should add more reasonable information verification. #2079

jianghuazhu · 2020-06-17T11:59:15Z

… is not perfect.

NOTICE

Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

hadoop-yetus · 2020-06-17T15:42:28Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	27m 18s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1 💚	mvninstall	22m 13s	trunk passed
+1 💚	compile	1m 10s	trunk passed
+1 💚	checkstyle	0m 47s	trunk passed
+1 💚	mvnsite	1m 14s	trunk passed
+1 💚	shadedclient	17m 17s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 45s	trunk passed
+0 🆗	spotbugs	3m 8s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 7s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	1m 11s	the patch passed
+1 💚	compile	1m 5s	the patch passed
+1 💚	javac	1m 5s	the patch passed
+1 💚	checkstyle	0m 44s	the patch passed
+1 💚	mvnsite	1m 10s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	15m 46s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 41s	the patch passed
+1 💚	findbugs	3m 11s	the patch passed
		_ Other Tests _
-1 ❌	unit	116m 57s	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 41s	The patch does not generate ASF License warnings.
		216m 30s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier
	hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
	hadoop.hdfs.TestReconstructStripedFile
	hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics
	hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate
	hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2079/1/artifact/out/Dockerfile
GITHUB PR	#2079
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 7d14130e7c57 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `c8ed33c`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-2079/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2079/1/testReport/
Max. process+thread count	2767 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2079/1/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2020-06-18T05:27:21Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 27s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1 💚	mvninstall	22m 37s	trunk passed
+1 💚	compile	1m 13s	trunk passed
+1 💚	checkstyle	0m 49s	trunk passed
+1 💚	mvnsite	1m 22s	trunk passed
+1 💚	shadedclient	18m 31s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 44s	trunk passed
+0 🆗	spotbugs	3m 1s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 59s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	1m 10s	the patch passed
+1 💚	compile	1m 2s	the patch passed
+1 💚	javac	1m 2s	the patch passed
+1 💚	checkstyle	0m 42s	the patch passed
+1 💚	mvnsite	1m 9s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	15m 37s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 39s	the patch passed
+1 💚	findbugs	3m 6s	the patch passed
		_ Other Tests _
-1 ❌	unit	116m 19s	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 34s	The patch does not generate ASF License warnings.
		191m 1s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
	hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier
	hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage
	hadoop.hdfs.TestReconstructStripedFile
	hadoop.hdfs.TestGetFileChecksum
	hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2079/2/artifact/out/Dockerfile
GITHUB PR	#2079
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 8bda8d92c4ff 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `caf3995`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-2079/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2079/2/testReport/
Max. process+thread count	2604 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2079/2/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

goiri · 2020-06-18T20:01:11Z

...fs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java

@@ -388,6 +388,11 @@ static int getParallelVolumeLoadThreadsNum(int dataDirs, Configuration conf) {
    try {
      final List<StorageLocation> successLocations = loadDataStorage(
          datanode, nsInfo, dataDirs, startOpt, executor);
+
+      if (successLocations.isEmpty()) {


Can we adjust the JIRA title to say something like this?
We should also have a test.

thank you very much.
I have modified the source code and resubmitted it.
The title has been modified.

…ble information verification.

hadoop-yetus · 2020-06-20T13:34:33Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 5s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	22m 18s	trunk passed
+1 💚	compile	1m 18s	trunk passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚	compile	1m 8s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	checkstyle	0m 46s	trunk passed
+1 💚	mvnsite	1m 13s	trunk passed
+1 💚	shadedclient	17m 12s	branch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 32s	hadoop-hdfs in trunk failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚	javadoc	0m 41s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+0 🆗	spotbugs	3m 7s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 4s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	1m 10s	the patch passed
+1 💚	compile	1m 9s	the patch passed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04
+1 💚	javac	1m 9s	the patch passed
+1 💚	compile	1m 4s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	javac	1m 4s	the patch passed
+1 💚	checkstyle	0m 39s	the patch passed
+1 💚	mvnsite	1m 9s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	15m 27s	patch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 30s	hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.
+1 💚	javadoc	0m 40s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	findbugs	3m 4s	the patch passed
		_ Other Tests _
-1 ❌	unit	107m 19s	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 35s	The patch does not generate ASF License warnings.
		183m 51s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy
	hadoop.hdfs.TestReconstructStripedFile
	hadoop.hdfs.server.namenode.TestDecommissioningStatus
	hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics
	hadoop.hdfs.TestStripedFileAppend

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2079/3/artifact/out/Dockerfile
GITHUB PR	#2079
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 50f8a5523e2d 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `b27810a`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2079/3/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-2079/3/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.7+10-post-Ubuntu-2ubuntu218.04.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-2079/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-2079/3/testReport/
Max. process+thread count	3254 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-2079/3/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

jianghuazhu force-pushed the HDFS-15416 branch from 950bcc5 to 52a42a6 Compare June 18, 2020 01:55

goiri reviewed Jun 18, 2020

View reviewed changes

HDFS-15416. DataStorage#addStorageLocations() should add more reasona…

01717a2

…ble information verification.

jianghuazhu force-pushed the HDFS-15416 branch from 52a42a6 to 01717a2 Compare June 20, 2020 10:14

jianghuazhu changed the title ~~HDFS-15416. The addStorageLocations() method in the DataStorage class…~~ HDFS-15416. DataStorage#addStorageLocations() should add more reasonable information verification. Jun 20, 2020

jianghuazhu closed this Jun 24, 2020

jianghuazhu deleted the HDFS-15416 branch June 24, 2020 02:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-15416. DataStorage#addStorageLocations() should add more reasonable information verification. #2079

HDFS-15416. DataStorage#addStorageLocations() should add more reasonable information verification. #2079

Uh oh!

jianghuazhu commented Jun 17, 2020

Uh oh!

hadoop-yetus commented Jun 17, 2020

Uh oh!

hadoop-yetus commented Jun 18, 2020

Uh oh!

goiri Jun 18, 2020

Uh oh!

jianghuazhu Jun 20, 2020

Uh oh!

hadoop-yetus commented Jun 20, 2020

Uh oh!

Uh oh!

HDFS-15416. DataStorage#addStorageLocations() should add more reasonable information verification. #2079

HDFS-15416. DataStorage#addStorageLocations() should add more reasonable information verification. #2079

Uh oh!

Conversation

jianghuazhu commented Jun 17, 2020

NOTICE

Uh oh!

hadoop-yetus commented Jun 17, 2020

Uh oh!

hadoop-yetus commented Jun 18, 2020

Uh oh!

goiri Jun 18, 2020

Choose a reason for hiding this comment

Uh oh!

jianghuazhu Jun 20, 2020

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Jun 20, 2020

Uh oh!

Uh oh!