HDFS-15781. Add metrics for how blocks are moved in replaceBlock. #2704

LeonGao91 · 2021-02-16T07:45:48Z

No description provided.

hadoop-yetus · 2021-02-16T12:54:19Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 9s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	34m 44s		trunk passed
+1 💚	compile	1m 20s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	1m 11s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	1m 1s		trunk passed
+1 💚	mvnsite	1m 20s		trunk passed
+1 💚	shadedclient	18m 4s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 53s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 23s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+0 🆗	spotbugs	3m 14s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 12s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 11s		the patch passed
+1 💚	compile	1m 14s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	1m 14s		the patch passed
+1 💚	compile	1m 6s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	1m 6s		the patch passed
-0 ⚠️	checkstyle	0m 55s	/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 237 unchanged - 0 fixed = 241 total (was 237)
+1 💚	mvnsite	1m 12s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	15m 17s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 49s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 17s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	findbugs	3m 20s		the patch passed
			_ Other Tests _
+1 💚	unit	214m 56s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		307m 25s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2704/1/artifact/out/Dockerfile
GITHUB PR	#2704
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 53bb4d0cae0b 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `2b3c5b1`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2704/1/testReport/
Max. process+thread count	2564 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2704/1/console
versions	git=2.25.1 maven=3.6.3 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-02-17T00:44:12Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 31s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 23s		trunk passed
+1 💚	compile	1m 19s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	1m 14s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	1m 8s		trunk passed
+1 💚	mvnsite	1m 19s		trunk passed
+1 💚	shadedclient	15m 30s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 54s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 22s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+0 🆗	spotbugs	3m 2s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 1s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 13s		the patch passed
+1 💚	compile	1m 11s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	1m 11s		the patch passed
+1 💚	compile	1m 4s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	1m 4s		the patch passed
+1 💚	checkstyle	1m 1s		the patch passed
+1 💚	mvnsite	1m 12s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	12m 44s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 49s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 18s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	findbugs	3m 4s		the patch passed
			_ Other Tests _
+1 💚	unit	189m 38s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 46s		The patch does not generate ASF License warnings.
		274m 26s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2704/2/artifact/out/Dockerfile
GITHUB PR	#2704
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 03aae445ad1b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `9882a22`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2704/2/testReport/
Max. process+thread count	3536 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2704/2/console
versions	git=2.25.1 maven=3.6.3 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Jing9 · 2021-02-18T23:19:27Z

...adoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java

+  private MutableCounterLong replaceBlockOpOnSameHostWithCopy;
+  @Metric("Number of replaceBlock ops between" +
+      " storage types on same disk mount using hardlink")
+  private MutableCounterLong replaceBlockOpOnSameHostWithHardlink;


Both "withHardlink" and "withCopy" are our block movement implementation. If in the future we change our implementation these names may no long hold. How about we change the metric names to "OnSameHost" and "OnSameMount" ? But then we need to think more about their semantic meanings. Maybe "OnSameHost" also includes the "OnSameMount"... Thoughts?

Yeah sounds good, OnSameHost can include OnSameMount actually. I will make the change.

hadoop-yetus · 2021-02-19T06:25:07Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 34s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 44s		trunk passed
+1 💚	compile	1m 18s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	1m 14s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	1m 6s		trunk passed
+1 💚	mvnsite	1m 20s		trunk passed
+1 💚	shadedclient	15m 56s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 54s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 26s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+0 🆗	spotbugs	21m 23s		Both FindBugs and SpotBugs are enabled, using SpotBugs.
+1 💚	spotbugs	3m 7s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 9s		the patch passed
+1 💚	compile	1m 11s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	1m 11s		the patch passed
+1 💚	compile	1m 5s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	1m 5s		the patch passed
+1 💚	checkstyle	0m 59s		the patch passed
+1 💚	mvnsite	1m 13s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	12m 54s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 48s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 23s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
-1 ❌	spotbugs	3m 2s	/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs cannot run computeBugHistory from spotbugs
			_ Other Tests _
-1 ❌	unit	196m 10s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 42s		The patch does not generate ASF License warnings.
		282m 5s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
	hadoop.hdfs.TestDecommissionWithStriped
	hadoop.hdfs.server.namenode.TestDecommissioningStatus

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2704/3/artifact/out/Dockerfile
GITHUB PR	#2704
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle
uname	Linux 5aa96703b036 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `2970bd9`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2704/3/testReport/
Max. process+thread count	3652 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2704/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-02-20T00:40:21Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 31s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 14s		trunk passed
+1 💚	compile	1m 21s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	1m 11s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	1m 4s		trunk passed
+1 💚	mvnsite	1m 21s		trunk passed
+1 💚	shadedclient	17m 46s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 54s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 22s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+0 🆗	spotbugs	3m 14s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	3m 11s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 12s		the patch passed
+1 💚	compile	1m 14s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	1m 14s		the patch passed
+1 💚	compile	1m 6s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	1m 6s		the patch passed
+1 💚	checkstyle	0m 59s		the patch passed
+1 💚	mvnsite	1m 12s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	15m 17s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 49s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 22s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	findbugs	3m 16s		the patch passed
			_ Other Tests _
-1 ❌	unit	214m 40s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 38s		The patch does not generate ASF License warnings.
		308m 8s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestRollingUpgrade
	hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2704/4/artifact/out/Dockerfile
GITHUB PR	#2704
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 484da49dbdae 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `d28b6f9`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2704/4/testReport/
Max. process+thread count	2617 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2704/4/console
versions	git=2.25.1 maven=3.6.3 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Jing9 · 2021-02-23T21:09:16Z

The current change looks good to me. +1

@LeonGao91 Could you please re-trigger the build to pass the checks?

LeonGao91 · 2021-02-24T06:25:44Z

Thanks @Jing9 ! Build is good now.

…ache#2704)

Add metrics for how blocks are moved in replaceBlock

37fb343

LeonGao91 added 2 commits February 16, 2021 11:13

Style

34b0459

Use better name

2d703f1

Jing9 reviewed Feb 18, 2021

View reviewed changes

Resolve comments

ea98c77

Trigger Build

b560f89

Trigger Build

ae55472

Jing9 merged commit 940c780 into apache:trunk Feb 24, 2021

kiran-maturi pushed a commit to kiran-maturi/hadoop that referenced this pull request Nov 24, 2021

HDFS-15781. Add metrics for how blocks are moved in replaceBlock. (ap…

28b8798

…ache#2704)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-15781. Add metrics for how blocks are moved in replaceBlock. #2704

HDFS-15781. Add metrics for how blocks are moved in replaceBlock. #2704

Uh oh!

LeonGao91 commented Feb 16, 2021

Uh oh!

hadoop-yetus commented Feb 16, 2021

Uh oh!

hadoop-yetus commented Feb 17, 2021

Uh oh!

Jing9 Feb 18, 2021

Uh oh!

LeonGao91 Feb 19, 2021

Uh oh!

hadoop-yetus commented Feb 19, 2021

Uh oh!

hadoop-yetus commented Feb 20, 2021

Uh oh!

Jing9 commented Feb 23, 2021

Uh oh!

LeonGao91 commented Feb 24, 2021

Uh oh!

Uh oh!

HDFS-15781. Add metrics for how blocks are moved in replaceBlock. #2704

HDFS-15781. Add metrics for how blocks are moved in replaceBlock. #2704

Uh oh!

Conversation

LeonGao91 commented Feb 16, 2021

Uh oh!

hadoop-yetus commented Feb 16, 2021

Uh oh!

hadoop-yetus commented Feb 17, 2021

Uh oh!

Jing9 Feb 18, 2021

Choose a reason for hiding this comment

Uh oh!

LeonGao91 Feb 19, 2021

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Feb 19, 2021

Uh oh!

hadoop-yetus commented Feb 20, 2021

Uh oh!

Jing9 commented Feb 23, 2021

Uh oh!

LeonGao91 commented Feb 24, 2021

Uh oh!

Uh oh!