HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. #3200

amahussein · 2021-07-13T16:02:21Z

Backporting HDFS-15160 to branch-3.2
- First: HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell
  - git cherry-pick -x d7c136b9ed6d99e1b03f5b89723b3a20df359ba8
  - main conflicts:
    - FsDatasetImpl.java.deepCopyReplica
    - FoldedTreeSet is not used in branch-3.2
  - testing:
    - mvn test -Dtest=TestWriteToReplica,TestReplicaMap,\ TestProvidedImpl,TestFsVolumeList,TestInterDatanodeProtocol
- Second: HDFS-15160
  - git cherry-pick -x 2a67e2b1a0e3a5f91056f5b977ef9c4c07ba6718
  - main conflicts:
    - ReplicaMap.java
    - FsDatasetImpl.java.deepCopyReplica
    - FsDatasetImpl.java.addVolume different code. I passed the red and write locks to tempVolumeMap
    - DirectoryScanner.scan() conflict in the entire loop.
      - HDFS-15415. Reduce locking in Datanode DirectoryScanner Which is already committed to branch-2.10 removes what HDFS-15160 has added. So, I skipped that.
- Third: HDFS-15457. TestFsDatasetImpl fails intermittently (#2407) and HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig (#2679)
  - git cherry-pick -x 98097b8f19789605b9697f6a959da57261e0fe19 9434c1eccc255a25ea5e11f6d8c9e1f83996d6b4
  - main conflicts:
    - Java imports

hadoop-yetus · 2021-08-08T08:20:15Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 18s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 9 new or modified test files.
			_ branch-3.2 Compile Tests _
+0 🆗	mvndep	3m 24s		Maven dependency ordering for branch
+1 💚	mvninstall	25m 29s		branch-3.2 passed
+1 💚	compile	15m 44s		branch-3.2 passed
+1 💚	checkstyle	2m 42s		branch-3.2 passed
+1 💚	mvnsite	2m 35s		branch-3.2 passed
+1 💚	javadoc	2m 14s		branch-3.2 passed
+1 💚	spotbugs	5m 10s		branch-3.2 passed
+1 💚	shadedclient	16m 45s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 21s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 58s		the patch passed
+1 💚	compile	15m 5s		the patch passed
-1 ❌	javac	15m 5s	/results-compile-javac-root.txt	root generated 1 new + 1587 unchanged - 1 fixed = 1588 total (was 1588)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	2m 39s		root: The patch generated 0 new + 569 unchanged - 1 fixed = 569 total (was 570)
+1 💚	mvnsite	2m 33s		the patch passed
+1 💚	xml	0m 2s		The patch has no ill-formed XML file.
+1 💚	javadoc	2m 4s		the patch passed
+1 💚	spotbugs	5m 26s		the patch passed
+1 💚	shadedclient	16m 49s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	15m 48s		hadoop-common in the patch passed.
-1 ❌	unit	201m 29s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
+1 💚	asflicense	0m 53s		The patch does not generate ASF License warnings.
		338m 29s

Reason	Tests
Failed junit tests	hadoop.hdfs.qjournal.server.TestJournalNodeSync

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/2/artifact/out/Dockerfile
GITHUB PR	#3200
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux 7941b9e37b6c 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.2 / 9516ddf5488d512bd55f539363b0befe575b893a
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/2/testReport/
Max. process+thread count	2331 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/2/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Hexiaoqiao · 2021-08-10T10:48:09Z

Thanks @amahussein for your works.
cc @sodonnel would you mind to give another checks? Consider branch-3.2.3 is prepared. I think it is good choice to involve this improvement there.

hadoop-yetus · 2021-08-25T08:42:07Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 0s		Docker mode activated.
-1 ❌	patch	0m 23s		HDFS-15160 does not apply to branch-3.3. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.

Subsystem	Report/Notes
JIRA Issue	HDFS-15160
JIRA Patch URL	https://issues.apache.org/jira/secure/attachment/13022920/HDFS-15160-branch-3.3-001.patch
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/3/console
versions	git=2.17.1
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-08-25T14:58:33Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	11m 10s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 9 new or modified test files.
			_ branch-3.2 Compile Tests _
+0 🆗	mvndep	3m 18s		Maven dependency ordering for branch
+1 💚	mvninstall	25m 29s		branch-3.2 passed
+1 💚	compile	15m 28s		branch-3.2 passed
+1 💚	checkstyle	2m 42s		branch-3.2 passed
+1 💚	mvnsite	2m 32s		branch-3.2 passed
+1 💚	javadoc	2m 8s		branch-3.2 passed
+1 💚	spotbugs	5m 8s		branch-3.2 passed
+1 💚	shadedclient	16m 42s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 20s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 56s		the patch passed
+1 💚	compile	14m 55s		the patch passed
-1 ❌	javac	14m 55s	/results-compile-javac-root.txt	root generated 1 new + 1587 unchanged - 1 fixed = 1588 total (was 1588)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	2m 37s		root: The patch generated 0 new + 568 unchanged - 1 fixed = 568 total (was 569)
+1 💚	mvnsite	2m 28s		the patch passed
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	2m 1s		the patch passed
+1 💚	spotbugs	5m 27s		the patch passed
+1 💚	shadedclient	16m 40s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	15m 41s		hadoop-common in the patch passed.
-1 ❌	unit	198m 18s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
+1 💚	asflicense	0m 53s		The patch does not generate ASF License warnings.
		343m 54s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.TestNameNodeMXBean

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/4/artifact/out/Dockerfile
GITHUB PR	#3200
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux 25db101e175e 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.2 / 9516ddf5488d512bd55f539363b0befe575b893a
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/4/testReport/
Max. process+thread count	1991 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/4/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Hexiaoqiao · 2021-08-26T03:18:22Z

Thanks @amahussein for your works. I trigger Yetus and it seems works well. Failed unit test TestDirectoryScanner looks not related to changes. AND verify this feature with pseudo cluster, it also seems work well.
Just notice that this PR include different cherry-pick commits together. I am not sure if it is traced gracefully when check in.
@brahmareddybattula do you have any idea here? If not I would like give my +1.

Hexiaoqiao · 2021-08-26T03:52:16Z

@jojochuang Would you mind to have another check?

hadoop-yetus · 2021-08-31T10:24:46Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	11m 44s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 9 new or modified test files.
			_ branch-3.2 Compile Tests _
+0 🆗	mvndep	4m 22s		Maven dependency ordering for branch
+1 💚	mvninstall	25m 31s		branch-3.2 passed
+1 💚	compile	15m 31s		branch-3.2 passed
+1 💚	checkstyle	2m 39s		branch-3.2 passed
+1 💚	mvnsite	2m 30s		branch-3.2 passed
+1 💚	javadoc	2m 6s		branch-3.2 passed
+1 💚	spotbugs	5m 3s		branch-3.2 passed
+1 💚	shadedclient	16m 34s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 21s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 55s		the patch passed
+1 💚	compile	14m 53s		the patch passed
-1 ❌	javac	14m 53s	/results-compile-javac-root.txt	root generated 1 new + 1576 unchanged - 1 fixed = 1577 total (was 1577)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	2m 42s		root: The patch generated 0 new + 568 unchanged - 1 fixed = 568 total (was 569)
+1 💚	mvnsite	2m 29s		the patch passed
+1 💚	xml	0m 2s		The patch has no ill-formed XML file.
+1 💚	javadoc	2m 2s		the patch passed
+1 💚	spotbugs	5m 26s		the patch passed
+1 💚	shadedclient	16m 42s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	15m 45s		hadoop-common in the patch passed.
-1 ❌	unit	208m 36s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
+1 💚	asflicense	1m 39s		The patch does not generate ASF License warnings.
		357m 8s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestReconstructStripedFileWithValidator
	hadoop.hdfs.TestDecommissionWithStriped
	hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/5/artifact/out/Dockerfile
GITHUB PR	#3200
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux 573c79676bd4 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.2 / 9516ddf5488d512bd55f539363b0befe575b893a
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/5/testReport/
Max. process+thread count	1948 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/5/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

amahussein · 2021-08-31T13:24:08Z

Thanks @amahussein for your works. I trigger Yetus and it seems works well. Failed unit test TestDirectoryScanner looks not related to changes. AND verify this feature with pseudo cluster, it also seems work well.
Just notice that this PR include different cherry-pick commits together. I am not sure if it is traced gracefully when check in.
@brahmareddybattula do you have any idea here? If not I would like give my +1.

Thanks @Hexiaoqiao for the review and the feedback. I appreciate it.

I intentionally cherry-picked the commits related to the feature so that the PR would be complete. Otherwise, if we did separate PRs for each commit, branch-3.2 would be broken.
I believe it should be okay to have multiple cherry-picks as long as the final merge would not squash them into one.

Hexiaoqiao · 2021-08-31T13:30:29Z

Thanks @amahussein. It is OK for me to squash and check in them together. Just find there are many checks failed reported by Yetus here. Do you mind give another check?
Considering branch-3.2 will release recently, I think it is time to push this PR into branch-3.2 before release happen.

amahussein · 2021-08-31T13:33:54Z

Thanks @amahussein. It is OK for me to squash and check in them together. Just find there are many checks failed reported by Yetus here. Do you mind give another check?
Considering branch-3.2 will release recently, I think it is time to push this PR into branch-3.2 before release happen.

I will rebase the code changes and push a fresh version.
P.S: the final merge "should not" squash the commits.

hadoop-yetus · 2021-08-31T19:40:21Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	8m 8s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 9 new or modified test files.
			_ branch-3.2 Compile Tests _
+0 🆗	mvndep	3m 50s		Maven dependency ordering for branch
+1 💚	mvninstall	24m 44s		branch-3.2 passed
+1 💚	compile	16m 34s		branch-3.2 passed
+1 💚	checkstyle	2m 28s		branch-3.2 passed
+1 💚	mvnsite	2m 40s		branch-3.2 passed
+1 💚	javadoc	2m 16s		branch-3.2 passed
+1 💚	spotbugs	4m 59s		branch-3.2 passed
+1 💚	shadedclient	14m 51s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 23s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 57s		the patch passed
+1 💚	compile	14m 24s		the patch passed
-1 ❌	javac	14m 24s	/results-compile-javac-root.txt	root generated 1 new + 1587 unchanged - 1 fixed = 1588 total (was 1588)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	2m 33s		root: The patch generated 0 new + 569 unchanged - 1 fixed = 569 total (was 570)
+1 💚	mvnsite	2m 44s		the patch passed
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	2m 19s		the patch passed
+1 💚	spotbugs	5m 22s		the patch passed
+1 💚	shadedclient	15m 20s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	15m 54s		hadoop-common in the patch passed.
-1 ❌	unit	174m 57s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
+1 💚	asflicense	1m 3s		The patch does not generate ASF License warnings.
		316m 39s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.ha.TestHAAppend
	hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/6/artifact/out/Dockerfile
GITHUB PR	#3200
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux 8ac03118cb9f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.2 / 29a98320b76d60afb481f729456fbb9db1c37287
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/6/testReport/
Max. process+thread count	3189 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/6/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

amahussein · 2021-08-31T21:02:07Z

After rebasing, it looks like there are javac checks added to the build that failed it.
Although this is a part of the original code, I decided to address it so that we can pass the build and move forward with the PR.
I also tested the TestDirectoryScanner and It passed.

hadoop-yetus · 2021-09-01T02:07:28Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 43s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 9 new or modified test files.
			_ branch-3.2 Compile Tests _
+0 🆗	mvndep	3m 27s		Maven dependency ordering for branch
+1 💚	mvninstall	22m 58s		branch-3.2 passed
+1 💚	compile	15m 2s		branch-3.2 passed
+1 💚	checkstyle	2m 36s		branch-3.2 passed
+1 💚	mvnsite	2m 42s		branch-3.2 passed
+1 💚	javadoc	2m 22s		branch-3.2 passed
+1 💚	spotbugs	5m 2s		branch-3.2 passed
+1 💚	shadedclient	15m 20s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 24s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 54s		the patch passed
+1 💚	compile	14m 21s		the patch passed
-1 ❌	javac	14m 21s	/results-compile-javac-root.txt	root generated 1 new + 1587 unchanged - 1 fixed = 1588 total (was 1588)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	2m 32s		root: The patch generated 0 new + 569 unchanged - 1 fixed = 569 total (was 570)
+1 💚	mvnsite	2m 41s		the patch passed
+1 💚	xml	0m 2s		The patch has no ill-formed XML file.
+1 💚	javadoc	2m 17s		the patch passed
+1 💚	spotbugs	5m 23s		the patch passed
+1 💚	shadedclient	15m 30s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	15m 57s		hadoop-common in the patch passed.
-1 ❌	unit	175m 41s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
+1 💚	asflicense	1m 4s		The patch does not generate ASF License warnings.
		306m 53s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.TestRedudantBlocks
	hadoop.hdfs.server.namenode.TestFsck

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/7/artifact/out/Dockerfile
GITHUB PR	#3200
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux aa126cf10a5a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.2 / 610e6e46ec87d615c19a51cf979bf43721706e07
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/7/testReport/
Max. process+thread count	3439 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/7/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

…n O'Donnell. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> (cherry picked from commit d7c136b)

…FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> (cherry picked from commit 2a67e2b)

(cherry picked from commit 98097b8)

…Contributed by Leon Gao (apache#2679) (cherry picked from commit 9434c1e)

hadoop-yetus · 2021-09-01T20:52:22Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 44s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 9 new or modified test files.
			_ branch-3.2 Compile Tests _
+0 🆗	mvndep	3m 35s		Maven dependency ordering for branch
+1 💚	mvninstall	24m 4s		branch-3.2 passed
+1 💚	compile	16m 26s		branch-3.2 passed
+1 💚	checkstyle	2m 43s		branch-3.2 passed
+1 💚	mvnsite	2m 49s		branch-3.2 passed
+1 💚	javadoc	2m 18s		branch-3.2 passed
+1 💚	spotbugs	5m 35s		branch-3.2 passed
+1 💚	shadedclient	17m 14s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 27s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 6s		the patch passed
+1 💚	compile	16m 37s		the patch passed
-1 ❌	javac	16m 37s	/results-compile-javac-root.txt	root generated 1 new + 1587 unchanged - 1 fixed = 1588 total (was 1588)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	2m 27s		root: The patch generated 0 new + 569 unchanged - 1 fixed = 569 total (was 570)
+1 💚	mvnsite	2m 45s		the patch passed
+1 💚	xml	0m 2s		The patch has no ill-formed XML file.
+1 💚	javadoc	2m 20s		the patch passed
+1 💚	spotbugs	5m 52s		the patch passed
+1 💚	shadedclient	17m 4s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	16m 31s	/patch-unit-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch failed.
-1 ❌	unit	181m 22s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch failed.
+1 💚	asflicense	1m 2s		The patch does not generate ASF License warnings.
		323m 19s

Reason	Tests
Failed junit tests	hadoop.ha.TestZKFailoverController
	hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
	hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
	hadoop.hdfs.server.namenode.TestFsck

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/8/artifact/out/Dockerfile
GITHUB PR	#3200
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux 12f0180fa72b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.2 / `7bfebda`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/8/testReport/
Max. process+thread count	3299 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3200/8/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Hexiaoqiao

LGTM. +1 from my side. Thanks @amahussein
Failed unit tests seems not related to this changes. Would like to wait other guys continue comments before check in.

brahmareddybattula · 2021-09-06T05:38:37Z

@amahussein thanks for working on this.. At first glance,it's better to have seperate PR for Each jira backport.It can be easy to review and maintain in future.

brahmareddybattula

HDFS-15818 changes lgtm.

brahmareddybattula

HDFS-15150 changes also lgtm.

brahmareddybattula

HDFS-15160 changes also lgtm.

brahmareddybattula

HDFS-15457 changes lgtm

brahmareddybattula · 2021-09-06T06:27:42Z

...-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java

  @Override
  public Set<? extends Replica> deepCopyReplica(String bpid)
      throws IOException {
-    Set<? extends Replica> replicas = null;
+    Set<? extends Replica> replicas;


can this change relate to trunk also..? Even this can be emptyset

Thanks @brahmareddybattula
I think we can apply some changes to trunk given that the type casting is unchecked and causes some Javac warnings.
Perhaps we can file another refactoring jira to address these issues on trunk.

can please raise jira to track this..?

amahussein · 2021-09-09T13:47:20Z

Hi @Hexiaoqiao, Is this PR ready for merge?

Hexiaoqiao · 2021-09-10T03:11:26Z

Will commit to branch-3.2 for a short while if no more comments.

Hexiaoqiao · 2021-09-11T11:53:01Z

Committed to branch-3.2. Thanks @amahussein for your works and thanks @brahmareddybattula for your reviews!

brahmareddybattula · 2021-09-13T19:48:22Z

@Hexiaoqiao , There are two issues I want to bring with this merge. Can you please check once.?

Looks commit's are squashed and merged, this should n't good way as there are 5 differenet commits with 5 jira's..In furute,if we want to revert one commit(among these 5), then all commits we need to revert.
It's not merged to branch-3.2.3, But jira marked to hadoop-3.2.3 version..?

Hexiaoqiao · 2021-09-14T04:34:54Z

@brahmareddybattula Good point. Will revert and check in per-commit. Thanks. For version, I forgot we have create branch-3.2.3 already.

… various FsDatasetImpl methods should use datanode readlock. (#3200)" This reverts commit 318bc5f.

Hexiaoqiao · 2021-09-14T06:22:52Z

revert squashed commit and cherry-pick single commit for branch-3.2.

Hexiaoqiao · 2021-09-14T07:18:51Z

Committed to branch-3.2.3.
cherry-pick to branch-3.2.3, build at local and run related unit tests passed.

aajisaka added the backport Backport PR; reviews may be optional depending on nature of backport label Aug 8, 2021

Hexiaoqiao requested review from brahmareddybattula, sodonnel, jojochuang, dineshchitlangia and ericbadger August 10, 2021 10:45

amahussein force-pushed the hdfs-15160-320 branch from 9516ddf to 29a9832 Compare August 31, 2021 14:22

Stephen O'Donnell and others added 5 commits September 1, 2021 09:18

HDFS-15150. Introduce read write lock to Datanode. Contributed Stephe…

78f395c

…n O'Donnell. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> (cherry picked from commit d7c136b)

HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various …

3c2285a

…FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell. Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org> (cherry picked from commit 2a67e2b)

HDFS-15457. TestFsDatasetImpl fails intermittently (apache#2407)

dcaccc1

(cherry picked from commit 98097b8)

HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. …

0c20e27

…Contributed by Leon Gao (apache#2679) (cherry picked from commit 9434c1e)

HDFS-15160. amend to fix javac error supressing unchecked warning

7bfebda

amahussein force-pushed the hdfs-15160-320 branch from 610e6e4 to 7bfebda Compare September 1, 2021 15:27

Hexiaoqiao approved these changes Sep 2, 2021

View reviewed changes

brahmareddybattula reviewed Sep 6, 2021

View reviewed changes

Hexiaoqiao merged commit 318bc5f into apache:branch-3.2 Sep 11, 2021

Hexiaoqiao added a commit that referenced this pull request Sep 14, 2021

Revert " HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and…

21fc30c

… various FsDatasetImpl methods should use datanode readlock. (#3200)" This reverts commit 318bc5f.

asfgit pushed a commit that referenced this pull request Sep 14, 2021

Revert " HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and…

ca182b6

… various FsDatasetImpl methods should use datanode readlock. (#3200)" This reverts commit 318bc5f.

HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. #3200

HDFS-15160. branch-3.2. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. #3200

Uh oh!

Conversation

amahussein commented Jul 13, 2021

Uh oh!

hadoop-yetus commented Aug 8, 2021

Uh oh!

Hexiaoqiao commented Aug 10, 2021

Uh oh!

hadoop-yetus commented Aug 25, 2021

Uh oh!

hadoop-yetus commented Aug 25, 2021

Uh oh!

Hexiaoqiao commented Aug 26, 2021

Uh oh!

Hexiaoqiao commented Aug 26, 2021

Uh oh!

hadoop-yetus commented Aug 31, 2021

Uh oh!

amahussein commented Aug 31, 2021

Uh oh!

Hexiaoqiao commented Aug 31, 2021

Uh oh!

amahussein commented Aug 31, 2021

Uh oh!

hadoop-yetus commented Aug 31, 2021

Uh oh!

amahussein commented Aug 31, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hadoop-yetus commented Sep 1, 2021

Uh oh!

hadoop-yetus commented Sep 1, 2021

Uh oh!

Hexiaoqiao left a comment

Choose a reason for hiding this comment

Uh oh!

brahmareddybattula commented Sep 6, 2021

Uh oh!

brahmareddybattula left a comment

Choose a reason for hiding this comment

Uh oh!

brahmareddybattula left a comment

Choose a reason for hiding this comment

Uh oh!

brahmareddybattula left a comment

Choose a reason for hiding this comment

Uh oh!

brahmareddybattula left a comment

Choose a reason for hiding this comment

Uh oh!

brahmareddybattula Sep 6, 2021

Choose a reason for hiding this comment

Uh oh!

amahussein Sep 7, 2021

Choose a reason for hiding this comment

Uh oh!

brahmareddybattula Sep 13, 2021

Choose a reason for hiding this comment

Uh oh!

amahussein commented Sep 9, 2021

Uh oh!

Hexiaoqiao commented Sep 10, 2021

Uh oh!

Hexiaoqiao commented Sep 11, 2021

Uh oh!

brahmareddybattula commented Sep 13, 2021

Uh oh!

Hexiaoqiao commented Sep 14, 2021

Uh oh!

Hexiaoqiao commented Sep 14, 2021

Uh oh!

Hexiaoqiao commented Sep 14, 2021

Uh oh!

Uh oh!

amahussein commented Aug 31, 2021 •

edited

Loading