HDFS-15842. HDFS mover to emit metrics. #2738

LeonGao91 · 2021-03-03T07:12:29Z

No description provided.

Jing9 · 2021-03-08T07:08:33Z

...hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/package-info.java

@@ -0,0 +1,27 @@
+/**


Do we need to add this file?

It is to pass the checkstyle as I added a new file to this package without package-info..

Jing9 · 2021-03-08T07:27:37Z

hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java

@@ -649,6 +664,11 @@ static int run(Map<URI, List<Path>> namenodes, Configuration conf)
    Map<Long, Set<DatanodeInfo>> excludedPinnedBlocks = new HashMap<>();
    LOG.info("namenodes = " + namenodes);

+    DefaultMetricsSystem.initialize("Mover");


Does Balancer have a similar metrics? If not shall we use this PR or create a new one to do that?

We have it in the internal branch with similar implementation but it is not in upstream yet.

Actually, there has been a JIRA long time ago for balancer metrics. https://issues.apache.org/jira/browse/HDFS-10648

I can rebase our changes there as a different PR after this.

hadoop-yetus · 2021-06-17T05:30:21Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 33s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	30m 45s		trunk passed
+1 💚	compile	1m 22s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 14s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 2s		trunk passed
+1 💚	mvnsite	1m 25s		trunk passed
+1 💚	javadoc	0m 57s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 27s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 9s		trunk passed
+1 💚	shadedclient	16m 25s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 12s		the patch passed
+1 💚	compile	1m 12s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	1m 12s		the patch passed
+1 💚	compile	1m 6s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	1m 6s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 53s		hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 100 unchanged - 1 fixed = 100 total (was 101)
+1 💚	mvnsite	1m 14s		the patch passed
+1 💚	javadoc	0m 47s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 20s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 8s		the patch passed
+1 💚	shadedclient	15m 51s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	237m 4s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 45s		The patch does not generate ASF License warnings.
		320m 52s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2738/3/artifact/out/Dockerfile
GITHUB PR	#2738
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 89ceb310f5a7 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `4dc6df6`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2738/3/testReport/
Max. process+thread count	3672 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2738/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Jing9

Thanks for updating the patch! The patch LGTM. Only one minor comment.

Jing9 · 2021-06-18T21:32:46Z

...hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/MoverMetrics.java

+
+  public static MoverMetrics create(Mover mover) {
+    MoverMetrics m = new MoverMetrics(mover);
+    DefaultMetricsSystem.instance().unregisterSource(m.getName());


Any reason we want to call unregister here? Can we call unregister at the end of the mover running?

You are right, this is not needed here. As discussed I will shutdown metrics at the end of the mover run

hadoop-yetus · 2021-06-19T05:49:28Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	13m 16s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	30m 53s		trunk passed
+1 💚	compile	1m 22s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	1m 19s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 3s		trunk passed
+1 💚	mvnsite	1m 26s		trunk passed
+1 💚	javadoc	0m 57s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 29s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 9s		trunk passed
+1 💚	shadedclient	16m 28s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 10s		the patch passed
+1 💚	compile	1m 12s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	1m 12s		the patch passed
+1 💚	compile	1m 10s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	1m 10s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 54s		hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 100 unchanged - 1 fixed = 100 total (was 101)
+1 💚	mvnsite	1m 16s		the patch passed
+1 💚	javadoc	0m 46s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 21s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	3m 9s		the patch passed
+1 💚	shadedclient	15m 57s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	230m 30s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 46s		The patch does not generate ASF License warnings.
		327m 18s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestRollingUpgrade

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2738/4/artifact/out/Dockerfile
GITHUB PR	#2738
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux cab7ae036721 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `96525f6`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2738/4/testReport/
Max. process+thread count	3829 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2738/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Jing9 · 2021-06-19T22:39:14Z

+1

Jing9 reviewed Mar 8, 2021

View reviewed changes

LeonGao91 added 2 commits June 16, 2021 17:07

Add metrics to mover

abec10d

Improve style

4dc6df6

LeonGao91 force-pushed the HDFS-15842-dev branch from 26469e2 to 4dc6df6 Compare June 17, 2021 00:08

Jing9 reviewed Jun 18, 2021

View reviewed changes

Resolve comment

96525f6

Jing9 merged commit 643dfd6 into apache:trunk Jun 19, 2021

kiran-maturi pushed a commit to kiran-maturi/hadoop that referenced this pull request Nov 24, 2021

HDFS-15842. HDFS mover to emit metrics. (apache#2738)

adc4360

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-15842. HDFS mover to emit metrics. #2738

HDFS-15842. HDFS mover to emit metrics. #2738

Uh oh!

LeonGao91 commented Mar 3, 2021

Uh oh!

Jing9 Mar 8, 2021

Uh oh!

LeonGao91 Mar 8, 2021

Uh oh!

Jing9 Mar 8, 2021

Uh oh!

LeonGao91 Mar 8, 2021 •

edited

Loading

Uh oh!

hadoop-yetus commented Jun 17, 2021

Uh oh!

Jing9 left a comment

Uh oh!

Jing9 Jun 18, 2021

Uh oh!

LeonGao91 Jun 19, 2021

Uh oh!

hadoop-yetus commented Jun 19, 2021

Uh oh!

Jing9 commented Jun 19, 2021

Uh oh!

Uh oh!

HDFS-15842. HDFS mover to emit metrics. #2738

HDFS-15842. HDFS mover to emit metrics. #2738

Uh oh!

Conversation

LeonGao91 commented Mar 3, 2021

Uh oh!

Jing9 Mar 8, 2021

Choose a reason for hiding this comment

Uh oh!

LeonGao91 Mar 8, 2021

Choose a reason for hiding this comment

Uh oh!

Jing9 Mar 8, 2021

Choose a reason for hiding this comment

Uh oh!

LeonGao91 Mar 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Jun 17, 2021

Uh oh!

Jing9 left a comment

Choose a reason for hiding this comment

Uh oh!

Jing9 Jun 18, 2021

Choose a reason for hiding this comment

Uh oh!

LeonGao91 Jun 19, 2021

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Jun 19, 2021

Uh oh!

Jing9 commented Jun 19, 2021

Uh oh!

Uh oh!

LeonGao91 Mar 8, 2021 •

edited

Loading