HDFS-17042 Add rpcCallSuccesses and OverallRpcProcessingTime to RpcMetrics for Namenode (#5730) #5804

xinglin · 2023-07-04T06:04:54Z

…

Description of PR

Backport of HDFS-17042 from trunk to branch-3.3. Almost clean cherry-pick with a small conflict in RpcMetrics.java (rpcRequeueCalls metric has not been backported to branch-3.3 yet. Removed from this backport.).

How was this patch tested?

mvn test -Dtest=TestRPC,TestMutableMetrics,TestProtoBufRpc
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.metrics2.lib.TestMutableMetrics
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.221 s - in org.apache.hadoop.metrics2.lib.TestMutableMetrics
[INFO] Running org.apache.hadoop.ipc.TestRPC
[INFO] Tests run: 34, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.383 s - in org.apache.hadoop.ipc.TestRPC
[INFO] Running org.apache.hadoop.ipc.TestProtoBufRpc
[WARNING] Tests run: 18, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 7.261 s - in org.apache.hadoop.ipc.TestProtoBufRpc
[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 63, Failures: 0, Errors: 0, Skipped: 8

…trics for Namenode (apache#5730)

hadoop-yetus · 2023-07-04T09:34:38Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	4m 6s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ branch-3.3 Compile Tests _
-1 ❌	mvninstall	48m 4s	/branch-mvninstall-root.txt	root in branch-3.3 failed.
+1 💚	compile	20m 13s		branch-3.3 passed
+1 💚	checkstyle	0m 58s		branch-3.3 passed
+1 💚	mvnsite	1m 46s		branch-3.3 passed
+1 💚	javadoc	1m 0s		branch-3.3 passed
+1 💚	spotbugs	2m 42s		branch-3.3 passed
+1 💚	shadedclient	42m 51s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 4s		the patch passed
+1 💚	compile	18m 33s		the patch passed
+1 💚	javac	18m 33s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 2s	/results-checkstyle-hadoop-common-project_hadoop-common.txt	hadoop-common-project/hadoop-common: The patch generated 2 new + 261 unchanged - 0 fixed = 263 total (was 261)
+1 💚	mvnsite	1m 48s		the patch passed
+1 💚	javadoc	0m 56s		the patch passed
+1 💚	spotbugs	2m 49s		the patch passed
+1 💚	shadedclient	39m 10s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 27s		hadoop-common in the patch passed.
+1 💚	asflicense	1m 11s		The patch does not generate ASF License warnings.
		208m 15s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5804/1/artifact/out/Dockerfile
GITHUB PR	#5804
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux d68b59bc22e8 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `14228ca`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~18.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5804/1/testReport/
Max. process+thread count	3159 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5804/1/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

xinglin · 2023-07-04T18:49:24Z

mvn install command failed, due to some transient network connection issue.

[ERROR] Plugin org.apache.maven.plugins:maven-failsafe-plugin:3.0.0-M1 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-failsafe-plugin:jar:3.0.0-M1: Could not transfer artifact org.apache.maven.plugins:maven-failsafe-plugin:pom:3.0.0-M1 from/to central (https://repo.maven.apache.org/maven2): Connection reset -> [Help 1]

Able to build locally.

 ~/p/h/branch-3.3 (HDFS-17042-branch-3.3)> mvn clean install -DskipTests
[INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [  0.467 s]
[INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [  0.204 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [  0.748 s]
[INFO] Apache Hadoop Client Modules ....................... SUCCESS [  0.029 s]
[INFO] Apache Hadoop Tencent COS Support .................. SUCCESS [  0.937 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [  0.521 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [  0.029 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  17:44 min
[INFO] Finished at: 2023-07-04T11:45:00-07:00

Made a couple of empty commits, to trigger multiple builds.

hadoop-yetus · 2023-07-04T22:09:55Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 36s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	49m 8s		branch-3.3 passed
+1 💚	compile	19m 24s		branch-3.3 passed
+1 💚	checkstyle	1m 7s		branch-3.3 passed
+1 💚	mvnsite	1m 50s		branch-3.3 passed
+1 💚	javadoc	1m 3s		branch-3.3 passed
+1 💚	spotbugs	2m 44s		branch-3.3 passed
+1 💚	shadedclient	39m 23s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 3s		the patch passed
+1 💚	compile	18m 29s		the patch passed
+1 💚	javac	18m 29s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 3s	/results-checkstyle-hadoop-common-project_hadoop-common.txt	hadoop-common-project/hadoop-common: The patch generated 2 new + 261 unchanged - 0 fixed = 263 total (was 261)
+1 💚	mvnsite	1m 48s		the patch passed
+1 💚	javadoc	0m 56s		the patch passed
+1 💚	spotbugs	2m 48s		the patch passed
+1 💚	shadedclient	39m 24s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 27s		hadoop-common in the patch passed.
+1 💚	asflicense	1m 10s		The patch does not generate ASF License warnings.
		201m 47s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5804/2/artifact/out/Dockerfile
GITHUB PR	#5804
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux 3f0ab92afc71 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `250b20e`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~18.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5804/2/testReport/
Max. process+thread count	3159 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5804/2/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

xinglin · 2023-07-05T00:02:34Z

The two checkstyle warnings are because we normally do not add private for metric variables (other metrics don't have visibility modifier neither).

RpcDetailedMetrics.java:46:  @Metric MutableRatesWithAggregation overallRpcProcessingRates;:39: Variable 'overallRpcProcessingRates' must be private and have accessor methods. [VisibilityModifier]
RpcMetrics.java:133:  MutableCounterLong rpcCallSuccesses;:22: Variable 'rpcCallSuccesses' must be private and have accessor methods. [VisibilityModifier]

@goiri, could you help merge this PR into branch-3.3 as well? thanks,

xinglin · 2023-07-17T23:43:42Z

@goiri, ping for review.

…trics for Namenode (apache#268) * HDFS-17042 Add rpcCallSuccesses and OverallRpcProcessingTime to RpcMetrics for Namenode (apache#5730) (apache#5804) --------- Co-authored-by: Xing Lin <linxingnku@gmail.com>

HDFS-17042 Add rpcCallSuccesses and OverallRpcProcessingTime to RpcMe…

14228ca

…trics for Namenode (apache#5730)

Xing Lin added 5 commits July 4, 2023 11:46

An empty commit to trigger a build

60edd75

An empty commit to trigger a build

d541a33

An empty commit to trigger a build

e9e5d04

An empty commit to trigger a build

d3352f5

An empty commit to trigger a build

250b20e

xinglin marked this pull request as ready for review July 4, 2023 23:57

goiri approved these changes Jul 19, 2023

View reviewed changes

goiri merged commit 8ee580a into apache:branch-3.3 Jul 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-17042 Add rpcCallSuccesses and OverallRpcProcessingTime to RpcMetrics for Namenode (#5730) #5804

HDFS-17042 Add rpcCallSuccesses and OverallRpcProcessingTime to RpcMetrics for Namenode (#5730) #5804

Uh oh!

xinglin commented Jul 4, 2023

Uh oh!

hadoop-yetus commented Jul 4, 2023

Uh oh!

xinglin commented Jul 4, 2023 •

edited

Loading

Uh oh!

hadoop-yetus commented Jul 4, 2023

Uh oh!

xinglin commented Jul 5, 2023

Uh oh!

xinglin commented Jul 17, 2023

Uh oh!

Uh oh!

HDFS-17042 Add rpcCallSuccesses and OverallRpcProcessingTime to RpcMetrics for Namenode (#5730) #5804

HDFS-17042 Add rpcCallSuccesses and OverallRpcProcessingTime to RpcMetrics for Namenode (#5730) #5804

Uh oh!

Conversation

xinglin commented Jul 4, 2023

Description of PR

How was this patch tested?

Uh oh!

hadoop-yetus commented Jul 4, 2023

Uh oh!

xinglin commented Jul 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hadoop-yetus commented Jul 4, 2023

Uh oh!

xinglin commented Jul 5, 2023

Uh oh!

xinglin commented Jul 17, 2023

Uh oh!

Uh oh!

xinglin commented Jul 4, 2023 •

edited

Loading