HDFS-16948 Update log of BlockManager#chooseExcessRedundancyStriped when EC internal block is moved by balancer. #5474

RuinanGu · 2023-03-12T17:49:42Z

Description of PR

This is a follow-up of HDFS-16179.When the EC internal block is moved by the balancer, it will trigger chooseExcessRedundancyStriped( ), and the parameter delNodeHint is not null.So the nonExcess list will be modified in processChosenExcessRedundancy( ), normally the size of nonExcess will be reduced to the expected EC group size, that's why the annoying log "excess types chose ... is empty." will be printed.

RuinanGu · 2023-03-12T18:25:24Z

@tomscut Could you please take a look?

hadoop-yetus · 2023-03-12T23:02:02Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 36s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	38m 56s		trunk passed
+1 💚	compile	1m 26s		trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	compile	1m 22s		trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	checkstyle	1m 8s		trunk passed
+1 💚	mvnsite	1m 30s		trunk passed
+1 💚	javadoc	1m 9s		trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	1m 34s		trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	spotbugs	3m 28s		trunk passed
+1 💚	shadedclient	22m 43s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 18s		the patch passed
+1 💚	compile	1m 16s		the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javac	1m 16s		the patch passed
+1 💚	compile	1m 14s		the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	javac	1m 14s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 52s		the patch passed
+1 💚	mvnsite	1m 19s		the patch passed
+1 💚	javadoc	0m 51s		the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	1m 28s		the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	spotbugs	3m 15s		the patch passed
+1 💚	shadedclient	22m 21s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	204m 10s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 49s		The patch does not generate ASF License warnings.
		311m 1s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5474/1/artifact/out/Dockerfile
GITHUB PR	#5474
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux d7ea53e842a1 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `c4224a7`
Default Java	Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5474/1/testReport/
Max. process+thread count	3617 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5474/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

zhangshuyan0 · 2023-03-14T07:49:29Z

Even if delStorageHint is not null, processChosenExcessRedundancy() may not be executed when delHint is not duplicated. I suggest adding a flag that is set to false after processChosenExcessRedundancy() is executed. Only log when the flag is true.

RuinanGu · 2023-03-14T16:49:31Z

@zhangshuyan0 Thank you, that makes sense.

Even if delStorageHint is not null, processChosenExcessRedundancy() may not be executed when delHint is not duplicated. I suggest adding a flag that is set to false after processChosenExcessRedundancy() is executed. Only log when the flag is true.

hadoop-yetus · 2023-03-14T21:59:31Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 41s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	39m 2s		trunk passed
+1 💚	compile	1m 18s		trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	compile	1m 21s		trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	checkstyle	1m 7s		trunk passed
+1 💚	mvnsite	1m 32s		trunk passed
+1 💚	javadoc	1m 8s		trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	1m 37s		trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	spotbugs	3m 26s		trunk passed
+1 💚	shadedclient	22m 36s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 17s		the patch passed
+1 💚	compile	1m 18s		the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javac	1m 18s		the patch passed
+1 💚	compile	1m 13s		the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	javac	1m 13s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 52s		the patch passed
+1 💚	mvnsite	1m 19s		the patch passed
+1 💚	javadoc	0m 50s		the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚	javadoc	1m 24s		the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚	spotbugs	3m 17s		the patch passed
+1 💚	shadedclient	22m 28s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	204m 16s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 50s		The patch does not generate ASF License warnings.
		311m 2s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5474/2/artifact/out/Dockerfile
GITHUB PR	#5474
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 7046b94e6ac7 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `8a015a9`
Default Java	Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5474/2/testReport/
Max. process+thread count	3146 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5474/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

zhangshuyan0 · 2023-03-16T02:01:58Z

@Hexiaoqiao Would you mind reviewing this?

Hexiaoqiao

+1, LGTM.

...ct/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java

Hexiaoqiao · 2023-03-20T12:20:58Z

@kidd53685368 Please check if the inline comment should fix. (just receive your review request but one review feedback not resolve yet). Thanks.

RuinanGu · 2023-03-21T08:46:53Z

@kidd53685368 Please check if the inline comment should fix. (just receive your review request but one review feedback not resolve yet). Thanks.

@Hexiaoqiao Thanks for the review.Already confirm it and I dont' think the inline comment should be fixed.

Hexiaoqiao

LGTM. +1.

Hexiaoqiao · 2023-03-23T09:01:13Z

Committed to trunk. Thanks @kidd53685368 for your contribution, and thanks @zhangshuyan0 for your reviews.

…when EC internal block is moved by balancer. (apache#5474). Contributed by Kidd53685368. Reviewed-by: zhangshuyan <zqingchai@gmail.com> Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>

Do not print log if delStorageHint is not null

c4224a7

RuinanGu changed the title ~~HDFS16948 Avoid too much logs in BlockManager#chooseExcessRedundancyStriped~~ HDFS-16948 Avoid too much logs in BlockManager#chooseExcessRedundancyStriped Mar 12, 2023

RuinanGu changed the title ~~HDFS-16948 Avoid too much logs in BlockManager#chooseExcessRedundancyStriped~~ HDFS-16948 Do not print "excess types chosen is empty" when EC internal block is moved by balancer Mar 12, 2023

RuinanGu changed the title ~~HDFS-16948 Do not print "excess types chosen is empty" when EC internal block is moved by balancer~~ HDFS-16948 Do not print "excess types chosen ... is empty" when EC internal block is moved by balancer Mar 12, 2023

Add a flag to determine to print log or not

8a015a9

zhangshuyan0 approved these changes Mar 15, 2023

View reviewed changes

Hexiaoqiao changed the title ~~HDFS-16948 Do not print "excess types chosen ... is empty" when EC internal block is moved by balancer~~ HDFS-16948 Update log of BlockManager#chooseExcessRedundancyStriped when EC internal block is moved by balancer. Mar 16, 2023

Hexiaoqiao approved these changes Mar 16, 2023

View reviewed changes

Hexiaoqiao reviewed Mar 16, 2023

View reviewed changes

...ct/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java Show resolved Hide resolved

RuinanGu requested a review from Hexiaoqiao March 20, 2023 11:24

Hexiaoqiao approved these changes Mar 23, 2023

View reviewed changes

Hexiaoqiao merged commit 5cf62d1 into apache:trunk Mar 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-16948 Update log of BlockManager#chooseExcessRedundancyStriped when EC internal block is moved by balancer. #5474

HDFS-16948 Update log of BlockManager#chooseExcessRedundancyStriped when EC internal block is moved by balancer. #5474

Uh oh!

RuinanGu commented Mar 12, 2023

Uh oh!

RuinanGu commented Mar 12, 2023

Uh oh!

hadoop-yetus commented Mar 12, 2023

Uh oh!

zhangshuyan0 commented Mar 14, 2023 •

edited

Loading

Uh oh!

RuinanGu commented Mar 14, 2023

Uh oh!

hadoop-yetus commented Mar 14, 2023

Uh oh!

zhangshuyan0 commented Mar 16, 2023

Uh oh!

Hexiaoqiao left a comment

Uh oh!

Uh oh!

Hexiaoqiao commented Mar 20, 2023

Uh oh!

RuinanGu commented Mar 21, 2023

Uh oh!

Hexiaoqiao left a comment

Uh oh!

Hexiaoqiao commented Mar 23, 2023

Uh oh!

Uh oh!

HDFS-16948 Update log of BlockManager#chooseExcessRedundancyStriped when EC internal block is moved by balancer. #5474

HDFS-16948 Update log of BlockManager#chooseExcessRedundancyStriped when EC internal block is moved by balancer. #5474

Uh oh!

Conversation

RuinanGu commented Mar 12, 2023

Description of PR

Uh oh!

RuinanGu commented Mar 12, 2023

Uh oh!

hadoop-yetus commented Mar 12, 2023

Uh oh!

zhangshuyan0 commented Mar 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RuinanGu commented Mar 14, 2023

Uh oh!

hadoop-yetus commented Mar 14, 2023

Uh oh!

zhangshuyan0 commented Mar 16, 2023

Uh oh!

Hexiaoqiao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Hexiaoqiao commented Mar 20, 2023

Uh oh!

RuinanGu commented Mar 21, 2023

Uh oh!

Hexiaoqiao left a comment

Choose a reason for hiding this comment

Uh oh!

Hexiaoqiao commented Mar 23, 2023

Uh oh!

Uh oh!

zhangshuyan0 commented Mar 14, 2023 •

edited

Loading