HDFS-16500. Make asynchronous blocks deletion lock and unlock durtion threshold configurable. #4061

smarthanwang · 2022-03-10T11:38:24Z

https://issues.apache.org/jira/browse/HDFS-16500

I have backport the nice feature HDFS-16043 to our internal branch, it works well in our testing cluster.

I think it's better to make the fields deleteBlockLockTimeMs and deleteBlockUnlockIntervalTimeMs configurable, so that we can control the lock and unlock duration.

private final long deleteBlockLockTimeMs = 500;
private final long deleteBlockUnlockIntervalTimeMs = 100;

And we should set the default value smaller to avoid blocking other requests long time when deleting some large directories.

… threshold configurable.

hadoop-yetus · 2022-03-10T16:44:17Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 39s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
-1 ❌	mvninstall	3m 32s	/branch-mvninstall-root.txt	root in trunk failed.
+1 💚	compile	2m 19s		trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 13s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 54s		trunk passed
+1 💚	mvnsite	1m 22s		trunk passed
+1 💚	javadoc	0m 55s		trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 26s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 9s		trunk passed
+1 💚	shadedclient	25m 21s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 15s		the patch passed
+1 💚	compile	1m 22s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	1m 22s		the patch passed
+1 💚	compile	1m 12s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 12s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 54s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 284 unchanged - 0 fixed = 285 total (was 284)
+1 💚	mvnsite	1m 21s		the patch passed
+1 💚	javadoc	0m 52s		the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 23s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 16s		the patch passed
+1 💚	shadedclient	22m 42s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	230m 30s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 46s		The patch does not generate ASF License warnings.
		303m 53s

Reason	Tests
Failed junit tests	hadoop.tools.TestHdfsConfigFields

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4061/1/artifact/out/Dockerfile
GITHUB PR	#4061
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux da3081bf758f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `4419c1c`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4061/1/testReport/
Max. process+thread count	3059 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4061/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

smarthanwang · 2022-03-11T02:36:05Z

@Hexiaoqiao could you please take a look?

zhuxiangyi · 2022-03-11T03:22:58Z

@smarthanwang Thanks for your work , new parameters need to be added to hdfs-default.xml.

smarthanwang · 2022-03-11T06:15:03Z

@smarthanwang Thanks for your work , new parameters need to be added to hdfs-default.xml.

@zhuxiangyi good suggestion, I would fix it.

Hexiaoqiao · 2022-03-11T13:25:05Z

Thanks @smarthanwang involve me here. It almost look good to me. Just one nit, it seems changes default value here, any consideration? Just suggest keep it if no any concerns. Thanks again.

hadoop-yetus · 2022-03-11T15:02:34Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 54s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	36m 9s		trunk passed
+1 💚	compile	1m 32s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	compile	1m 24s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 59s		trunk passed
+1 💚	mvnsite	1m 30s		trunk passed
+1 💚	javadoc	1m 4s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 30s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 24s		trunk passed
+1 💚	shadedclient	26m 52s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 24s		the patch passed
+1 💚	compile	1m 30s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javac	1m 30s		the patch passed
+1 💚	compile	1m 18s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	1m 18s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 52s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 284 unchanged - 0 fixed = 285 total (was 284)
+1 💚	mvnsite	1m 24s		the patch passed
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	0m 59s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 25s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	3m 36s		the patch passed
+1 💚	shadedclient	25m 23s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	348m 24s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		458m 59s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4061/2/artifact/out/Dockerfile
GITHUB PR	#4061
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml
uname	Linux ac02869061b7 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `19af2f3`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4061/2/testReport/
Max. process+thread count	2195 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4061/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

smarthanwang · 2022-03-14T02:27:04Z

Thanks @smarthanwang involve me here. It almost look good to me. Just one nit, it seems changes default value here, any consideration? Just suggest keep it if no any concerns. Thanks again.

@Hexiaoqiao thanks for comment.
When deleting a large directory, the deletion loop would continue for a long time, if we set the lock holding limit as 500ms, the rpcQueueTime of almost all rpc requests would increase to 500ms, it's too long for production cluster. So I think it's better to set a smaller default value which is not much greater than the avgRpcQueueTime of our cluster.

smarthanwang · 2022-03-23T08:49:05Z

Hi @Hexiaoqiao, do you have any suggestion?

Hexiaoqiao

Sorry for the late response. LGTM. +1 from my side.
Will commit if no other more comments.

smarthanwang · 2022-04-13T06:57:47Z

@Hexiaoqiao thanks for reviews.

Hexiaoqiao · 2022-04-21T04:36:29Z

Committed to trunk. Thanks @smarthanwang for your contributions.
BTW, Would you like to fix checkstyle about method length is 156 lines is over 150 limit for BlockManager, IMO this is historical issue, we should file another issue to refactor it.

smarthanwang · 2022-04-21T08:07:26Z

@Hexiaoqiao thanks you. I would fix the checkstyle later

smarthanwang · 2022-04-22T01:59:33Z

@Hexiaoqiao I have filed issue https://issues.apache.org/jira/browse/HDFS-16553 to fix the checkstyle, please take a look.

… threshold configurable. (apache#4061). Contributed by Chengwei Wang.

HDFS-16500. Make asynchronous blocks deletion lock and unlock durtion…

4419c1c

… threshold configurable.

Add default config values to hdfs-default.xml.

19af2f3

Hexiaoqiao approved these changes Apr 12, 2022

View reviewed changes

smarthanwang requested a review from Hexiaoqiao April 20, 2022 03:50

Hexiaoqiao approved these changes Apr 21, 2022

View reviewed changes

Hexiaoqiao changed the title ~~HDFS-16500. Make asynchronous blocks deletion lock and unlock durtion…~~ HDFS-16500. Make asynchronous blocks deletion lock and unlock durtion threshold configurable. Apr 21, 2022

Hexiaoqiao merged commit bf4730b into apache:trunk Apr 21, 2022

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

HDFS-16500. Make asynchronous blocks deletion lock and unlock durtion…

b385fed

… threshold configurable. (apache#4061). Contributed by Chengwei Wang.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-16500. Make asynchronous blocks deletion lock and unlock durtion threshold configurable. #4061

HDFS-16500. Make asynchronous blocks deletion lock and unlock durtion threshold configurable. #4061

smarthanwang commented Mar 10, 2022

hadoop-yetus commented Mar 10, 2022

smarthanwang commented Mar 11, 2022

zhuxiangyi commented Mar 11, 2022

smarthanwang commented Mar 11, 2022 •

edited

Loading

Hexiaoqiao commented Mar 11, 2022

hadoop-yetus commented Mar 11, 2022

smarthanwang commented Mar 14, 2022

smarthanwang commented Mar 23, 2022

Hexiaoqiao left a comment

smarthanwang commented Apr 13, 2022

Hexiaoqiao commented Apr 21, 2022

smarthanwang commented Apr 21, 2022

smarthanwang commented Apr 22, 2022

HDFS-16500. Make asynchronous blocks deletion lock and unlock durtion threshold configurable. #4061

HDFS-16500. Make asynchronous blocks deletion lock and unlock durtion threshold configurable. #4061

Conversation

smarthanwang commented Mar 10, 2022

hadoop-yetus commented Mar 10, 2022

smarthanwang commented Mar 11, 2022

zhuxiangyi commented Mar 11, 2022

smarthanwang commented Mar 11, 2022 • edited Loading

Hexiaoqiao commented Mar 11, 2022

hadoop-yetus commented Mar 11, 2022

smarthanwang commented Mar 14, 2022

smarthanwang commented Mar 23, 2022

Hexiaoqiao left a comment

Choose a reason for hiding this comment

smarthanwang commented Apr 13, 2022

Hexiaoqiao commented Apr 21, 2022

smarthanwang commented Apr 21, 2022

smarthanwang commented Apr 22, 2022

smarthanwang commented Mar 11, 2022 •

edited

Loading