HDFS-15160. branch-2.10 ReplicaMap, Disk Balancer, Directory Scanner …

[amahussein]

…and various FsDatasetImpl methods should use datanode readlock. Contributed By Stephen O'Donnell and Ahmed Hussein

Backporting HDFS-15160 ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock.

Also backported the two jiras that fix unit tests:

• HDFS-15818
• HDFS-15457

The main Conflicts were as follow:

• main conflicts caused by HDFS-10636. Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
  • FsDatasetImpl.java#validateBlockFile due to HDFS-10636
  • FsDatasetImpl.java#getFile only exists in branch-2.10 and was removed in HDFS-10636
  • DiskBalancer does not exist in branch-2.10
  • FsDatasetImpl.java#moveBlockAcrossStorage is different in branch-2.10

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 35s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ branch-2.10 Compile Tests _
+1 💚	mvninstall	14m 26s	branch-2.10 passed
+1 💚	compile	0m 58s	branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	compile	0m 50s	branch-2.10 passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10
+1 💚	checkstyle	0m 52s	branch-2.10 passed
+1 💚	mvnsite	1m 0s	branch-2.10 passed
+1 💚	javadoc	1m 15s	branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	javadoc	0m 50s	branch-2.10 passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10
+0 🆗	spotbugs	6m 30s	Both FindBugs and SpotBugs are enabled, using SpotBugs.
-1 ❌	spotbugs	2m 24s	hadoop-hdfs-project/hadoop-hdfs in branch-2.10 has 1 extant spotbugs warnings.
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 52s	the patch passed
+1 💚	compile	0m 52s	the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	javac	0m 52s	the patch passed
+1 💚	compile	0m 46s	the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10
+1 💚	javac	0m 46s	the patch passed
+1 💚	checkstyle	0m 46s	the patch passed
+1 💚	mvnsite	0m 54s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 2s	The patch has no ill-formed XML file.
+1 💚	javadoc	1m 8s	the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	javadoc	0m 44s	the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10
+1 💚	spotbugs	2m 29s	the patch passed
		_ Other Tests _
-1 ❌	unit	62m 31s	hadoop-hdfs in the patch failed.
+1 💚	asflicense	0m 33s	The patch does not generate ASF License warnings.
		99m 1s

Reason	Tests
Failed junit tests	hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys
	hadoop.hdfs.TestMultipleNNPortQOP

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3196/1/artifact/out/Dockerfile
GITHUB PR	#3196
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xml
uname	Linux 1ae77264dd0a 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-2.10 / 992d2c5
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10
Multi-JDK versions	/usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_262-b10 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10
spotbugs	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3196/1/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
unit	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3196/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3196/1/testReport/
Max. process+thread count	2825 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3196/1/console
versions	git=2.7.4 maven=3.3.9 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[amahussein]

https://issues.apache.org/jira/browse/HDFS-15762 describes the same error in hadoop.hdfs.TestMultipleNNPortQOP

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 55s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 2s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ branch-2.10 Compile Tests _
+1 💚	mvninstall	16m 36s		branch-2.10 passed
+1 💚	compile	1m 5s		branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	compile	0m 58s		branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
+1 💚	checkstyle	0m 35s		branch-2.10 passed
+1 💚	mvnsite	1m 5s		branch-2.10 passed
+1 💚	javadoc	1m 25s		branch-2.10 passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	javadoc	0m 54s		branch-2.10 passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
-1 ❌	spotbugs	2m 54s	/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html	hadoop-hdfs-project/hadoop-hdfs in branch-2.10 has 1 extant spotbugs warnings.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 57s		the patch passed
+1 💚	compile	0m 54s		the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	javac	0m 54s		the patch passed
+1 💚	compile	0m 48s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
+1 💚	javac	0m 48s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 29s		the patch passed
+1 💚	mvnsite	1m 4s		the patch passed
+1 💚	xml	0m 2s		The patch has no ill-formed XML file.
+1 💚	javadoc	1m 19s		the patch passed with JDK Azul Systems, Inc.-1.7.0_262-b10
+1 💚	javadoc	0m 51s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
+1 💚	spotbugs	3m 1s		the patch passed
			_ Other Tests _
-1 ❌	unit	79m 48s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 33s		The patch does not generate ASF License warnings.
		120m 18s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.blockmanagement.TestBlockTokenWithShortCircuitRead
	hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3196/1/artifact/out/Dockerfile
GITHUB PR	#3196
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux 85d75609290e 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-2.10 / ed183d5
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Multi-JDK versions	/usr/lib/jvm/zulu-7-amd64:Azul Systems, Inc.-1.7.0_262-b10 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3196/1/testReport/
Max. process+thread count	1999 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3196/1/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.