HDFS-15810. RBF: RBFMetrics's TotalCapacity out of bounds #2910

fengnanli · 2021-04-15T00:50:28Z

NOTICE

Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

...doop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/metrics/RBFMetrics.java

...-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/metrics/TestRBFMetrics.java

fengnanli · 2021-04-23T17:47:28Z

Would you update the web UI to use the new metrics?

hadoop/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/webapps/router/federationhealth.html

Lines 116 to 118 in 486ddb7

<tr><th>Total capacity</th><td>{TotalCapacity|fmt_bytes}</td></tr>

<tr><th>Used capacity</th><td>{UsedCapacity|fmt_bytes}</td></tr>

<tr><th>Remaining capacity</th><td>{RemainingCapacity|fmt_bytes}</td></tr>

Fixed.

fengnanli · 2021-04-27T00:27:45Z

@aajisaka @goiri @cxorm mind giving it another look? The test errors are irrelevant. I also attached a screenshot in the jira ticket for 2*LONG.max_value = 16ZB display capacity

...hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/metrics/TestMetricsBase.java

goiri · 2021-04-27T20:15:03Z

Can we align the name of the JIRA and the PR?

fengnanli · 2021-04-27T21:11:06Z

@goiri Updated the PR title to align with the JIRA title.

fengnanli · 2021-05-01T22:57:40Z

@aajisaka Can you take another look? Thanks!

...-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/metrics/TestRBFMetrics.java

hadoop-yetus · 2021-05-02T09:49:56Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 33s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	16m 4s		Maven dependency ordering for branch
+1 💚	mvninstall	20m 13s		trunk passed
+1 💚	compile	20m 47s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	18m 23s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	3m 43s		trunk passed
+1 💚	mvnsite	2m 22s		trunk passed
+1 💚	javadoc	1m 55s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	2m 38s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	3m 57s		trunk passed
+1 💚	shadedclient	14m 35s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 27s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 28s		the patch passed
+1 💚	compile	21m 47s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	21m 47s		the patch passed
+1 💚	compile	21m 39s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	21m 39s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	4m 1s		the patch passed
+1 💚	mvnsite	2m 24s		the patch passed
+1 💚	javadoc	1m 47s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	2m 44s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	4m 33s		the patch passed
+1 💚	shadedclient	16m 51s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 37s		hadoop-common in the patch passed.
-1 ❌	unit	19m 0s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt	hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	0m 51s		The patch does not generate ASF License warnings.
		223m 47s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.federation.router.TestRouterRpc
	hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2910/5/artifact/out/Dockerfile
GITHUB PR	#2910
Optional Tests	dupname asflicense mvnsite codespell markdownlint compile javac javadoc mvninstall unit shadedclient spotbugs checkstyle
uname	Linux 9c18113f0fbf 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `5e28574`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2910/5/testReport/
Max. process+thread count	2611 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2910/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

aajisaka

The test failures are not related to the patch: HDFS-15952

aajisaka · 2021-05-02T10:19:13Z

Thank you @fengnanli, @cxorm, and @goiri!

Reviewed-by: Inigo Goiri <inigoiri@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org> (cherry picked from commit 6e525ab)

Reviewed-by: Inigo Goiri <inigoiri@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

fengnanli changed the title ~~HDFS-15810 RBF: Use BigInteger for Router capacity metrics~~ HDFS-15810. RBF: Use BigInteger for Router capacity metrics Apr 15, 2021

This comment has been minimized.

Sign in to view

cxorm reviewed Apr 15, 2021

View reviewed changes

...doop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/metrics/RBFMetrics.java Outdated Show resolved Hide resolved

aajisaka reviewed Apr 15, 2021

View reviewed changes

...doop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/metrics/RBFMetrics.java Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

goiri reviewed Apr 19, 2021

View reviewed changes

...-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/metrics/TestRBFMetrics.java Show resolved Hide resolved

...-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/metrics/TestRBFMetrics.java Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

aajisaka reviewed Apr 27, 2021

View reviewed changes

...hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/metrics/TestMetricsBase.java Show resolved Hide resolved

fengnanli added 4 commits April 27, 2021 09:58

HDFS-15810 RBF:Use BigInteger for Router capacity metrics

067a004

Add new metrics in bigint instead of changing existing one

3a4e2e5

Comments

c76af4e

Comments

92264cd

This comment has been minimized.

Sign in to view

fengnanli changed the title ~~HDFS-15810. RBF: Use BigInteger for Router capacity metrics~~ HDFS-15810. RBF: RBFMetrics's TotalCapacity out of bounds Apr 27, 2021

goiri approved these changes Apr 28, 2021

View reviewed changes

aajisaka reviewed May 2, 2021

View reviewed changes

...-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/metrics/TestRBFMetrics.java Outdated Show resolved Hide resolved

Comments for tests

5e28574

fengnanli force-pushed the HDFS-15810 branch from 666eb74 to 5e28574 Compare May 2, 2021 06:04

aajisaka approved these changes May 2, 2021

View reviewed changes

aajisaka merged commit 6e525ab into apache:trunk May 2, 2021

aajisaka pushed a commit that referenced this pull request May 3, 2021

HDFS-15810. RBF: RBFMetrics's TotalCapacity out of bounds (#2910)

43fac73

Reviewed-by: Inigo Goiri <inigoiri@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org> (cherry picked from commit 6e525ab)

kiran-maturi pushed a commit to kiran-maturi/hadoop that referenced this pull request Nov 24, 2021

HDFS-15810. RBF: RBFMetrics's TotalCapacity out of bounds (apache#2910)

e6b4754

Reviewed-by: Inigo Goiri <inigoiri@apache.org> Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-15810. RBF: RBFMetrics's TotalCapacity out of bounds #2910

HDFS-15810. RBF: RBFMetrics's TotalCapacity out of bounds #2910

Uh oh!

fengnanli commented Apr 15, 2021

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

Uh oh!

Uh oh!

fengnanli commented Apr 23, 2021

Uh oh!

This comment has been minimized.

fengnanli commented Apr 27, 2021

Uh oh!

Uh oh!

goiri commented Apr 27, 2021

Uh oh!

This comment has been minimized.

fengnanli commented Apr 27, 2021

Uh oh!

fengnanli commented May 1, 2021

Uh oh!

Uh oh!

hadoop-yetus commented May 2, 2021

Uh oh!

aajisaka left a comment

Uh oh!

aajisaka commented May 2, 2021

Uh oh!

Uh oh!

HDFS-15810. RBF: RBFMetrics's TotalCapacity out of bounds #2910

HDFS-15810. RBF: RBFMetrics's TotalCapacity out of bounds #2910

Uh oh!

Conversation

fengnanli commented Apr 15, 2021

NOTICE

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

Uh oh!

Uh oh!

fengnanli commented Apr 23, 2021

Uh oh!

This comment has been minimized.

fengnanli commented Apr 27, 2021

Uh oh!

Uh oh!

goiri commented Apr 27, 2021

Uh oh!

This comment has been minimized.

fengnanli commented Apr 27, 2021

Uh oh!

fengnanli commented May 1, 2021

Uh oh!

Uh oh!

hadoop-yetus commented May 2, 2021

Uh oh!

aajisaka left a comment

Choose a reason for hiding this comment

Uh oh!

aajisaka commented May 2, 2021

Uh oh!

Uh oh!