HDFS-17055 Export HAState as a metric from Namenode for monitoring #5790

xinglin · 2023-06-27T19:10:35Z

Description of PR

Cherry-pick of 03902f5 from trunk, with minor conflicts on imports for a file.

How was this patch tested?

mvn test -Dtest=TestDFSRollback,TestDFSUpgrade,TestDFSFinalize
mvn test -Dtest="TestNameNodeMetricsLogger"
mvn test -Dtest="TestHAMetrics"

…pache#5764)

hadoop-yetus · 2023-06-28T01:24:31Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	7m 33s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 1s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	48m 14s		branch-3.3 passed
+1 💚	compile	1m 26s		branch-3.3 passed
+1 💚	checkstyle	0m 57s		branch-3.3 passed
+1 💚	mvnsite	1m 32s		branch-3.3 passed
+1 💚	javadoc	1m 43s		branch-3.3 passed
+1 💚	spotbugs	3m 18s		branch-3.3 passed
+1 💚	shadedclient	39m 38s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 19s		the patch passed
+1 💚	compile	1m 14s		the patch passed
+1 💚	javac	1m 14s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 46s		the patch passed
+1 💚	mvnsite	1m 21s		the patch passed
+1 💚	javadoc	1m 26s		the patch passed
+1 💚	spotbugs	3m 18s		the patch passed
+1 💚	shadedclient	38m 57s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	209m 9s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 56s		The patch does not generate ASF License warnings.
		362m 28s

Reason	Tests
Failed junit tests	hadoop.hdfs.qjournal.client.TestQuorumJournalManager
	hadoop.hdfs.TestMaintenanceState
	hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5790/4/artifact/out/Dockerfile
GITHUB PR	#5790
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 2172e0c84747 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `0e21b58`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~18.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5790/4/testReport/
Max. process+thread count	3982 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5790/4/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2023-06-28T01:54:01Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	13m 29s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	54m 46s		branch-3.3 passed
+1 💚	compile	1m 20s		branch-3.3 passed
+1 💚	checkstyle	0m 53s		branch-3.3 passed
+1 💚	mvnsite	1m 29s		branch-3.3 passed
+1 💚	javadoc	1m 38s		branch-3.3 passed
+1 💚	spotbugs	3m 24s		branch-3.3 passed
+1 💚	shadedclient	42m 44s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 21s		the patch passed
+1 💚	compile	1m 15s		the patch passed
+1 💚	javac	1m 15s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 45s		the patch passed
+1 💚	mvnsite	1m 22s		the patch passed
+1 💚	javadoc	1m 24s		the patch passed
+1 💚	spotbugs	3m 23s		the patch passed
+1 💚	shadedclient	42m 33s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	220m 24s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 45s		The patch does not generate ASF License warnings.
		392m 20s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5790/3/artifact/out/Dockerfile
GITHUB PR	#5790
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 89c446cda4fa 4.15.0-212-generic #223-Ubuntu SMP Tue May 23 13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `0e21b58`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~18.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5790/3/testReport/
Max. process+thread count	2306 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5790/3/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

xinglin · 2023-06-28T01:56:37Z

Hi @goiri,

There is a single code change commit in this PR and we have one clean build. Do you think we can merge now or should we try more empty commits to trigger more builds?

hadoop-yetus · 2023-06-28T02:03:21Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	5m 0s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	57m 56s		branch-3.3 passed
+1 💚	compile	1m 20s		branch-3.3 passed
+1 💚	checkstyle	0m 55s		branch-3.3 passed
+1 💚	mvnsite	1m 39s		branch-3.3 passed
+1 💚	javadoc	1m 39s		branch-3.3 passed
+1 💚	spotbugs	3m 59s		branch-3.3 passed
+1 💚	shadedclient	43m 47s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 25s		the patch passed
+1 💚	compile	1m 16s		the patch passed
+1 💚	javac	1m 16s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 46s		the patch passed
+1 💚	mvnsite	1m 33s		the patch passed
+1 💚	javadoc	1m 31s		the patch passed
+1 💚	spotbugs	3m 51s		the patch passed
+1 💚	shadedclient	43m 28s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	241m 47s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	1m 2s		The patch does not generate ASF License warnings.
		411m 44s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.namenode.TestNameNodeMXBean
	hadoop.hdfs.TestFileCreation
	hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
	hadoop.hdfs.TestDecommission
	hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier
	hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
	hadoop.hdfs.server.balancer.TestBalancer
	hadoop.hdfs.server.mover.TestMover
	hadoop.hdfs.TestRollingUpgrade
	hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5790/1/artifact/out/Dockerfile
GITHUB PR	#5790
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 0624dafcb419 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `9db3b74`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~18.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5790/1/testReport/
Max. process+thread count	2213 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5790/1/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2023-06-28T02:04:50Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 52s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ branch-3.3 Compile Tests _
+1 💚	mvninstall	56m 25s		branch-3.3 passed
+1 💚	compile	1m 28s		branch-3.3 passed
+1 💚	checkstyle	0m 59s		branch-3.3 passed
+1 💚	mvnsite	1m 41s		branch-3.3 passed
+1 💚	javadoc	1m 36s		branch-3.3 passed
+1 💚	spotbugs	3m 46s		branch-3.3 passed
+1 💚	shadedclient	43m 58s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 33s		the patch passed
+1 💚	compile	1m 23s		the patch passed
+1 💚	javac	1m 23s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 48s		the patch passed
+1 💚	mvnsite	1m 30s		the patch passed
+1 💚	javadoc	1m 26s		the patch passed
+1 💚	spotbugs	3m 46s		the patch passed
+1 💚	shadedclient	42m 28s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	240m 27s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 57s		The patch does not generate ASF License warnings.
		403m 51s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDecommissionWithBackoffMonitor
	hadoop.hdfs.server.namenode.TestFileTruncate
	hadoop.hdfs.server.mover.TestMover
	hadoop.hdfs.server.namenode.ha.TestEditLogTailer

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5790/2/artifact/out/Dockerfile
GITHUB PR	#5790
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 018fa1f5da57 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `0e21b58`
Default Java	Private Build-1.8.0_362-8u372-ga~~us1-0ubuntu1~~18.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5790/2/testReport/
Max. process+thread count	2211 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5790/2/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

xinglin · 2023-06-29T16:35:57Z

Hi @goiri ,

Could you help commit this PR into branch-3.3? It is the same change which you helped merge into trunk this Monday. This PR contains only one code-change commit and the rest are empty commits to trigger multiple builds. We have one clean build and that should be good enough to merge. If you have different thoughts, please let me know. thanks,

xinglin · 2023-07-03T16:51:25Z

Thanks @goiri for approving/merging this into branch-3.3! Happy July 4th!

…pache#5790) (apache#254) Co-authored-by: Xing Lin <linxingnku@gmail.com>

xinglin and others added 6 commits June 27, 2023 11:31

HDFS-17055 Export HAState as a metric from Namenode for monitoring (a…

46e178d

…pache#5764)

An empty commit to trigger a build

9db3b74

An empty commit to trigger a build

a6b93f9

An empty commit to trigger a build

9b561f1

An empty commit to trigger a build

7d09c5c

An empty commit to trigger a build

0e21b58

xinglin marked this pull request as ready for review June 28, 2023 01:54

goiri approved these changes Jul 3, 2023

View reviewed changes

goiri merged commit 6b9cfe4 into apache:branch-3.3 Jul 3, 2023

NyteKnight pushed a commit to NyteKnight/hadoop that referenced this pull request Jun 25, 2024

HDFS-17055 Export HAState as a metric from Namenode for monitoring (a…

d26cdec

…pache#5790) (apache#254) Co-authored-by: Xing Lin <linxingnku@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDFS-17055 Export HAState as a metric from Namenode for monitoring #5790

HDFS-17055 Export HAState as a metric from Namenode for monitoring #5790

Uh oh!

xinglin commented Jun 27, 2023

Uh oh!

hadoop-yetus commented Jun 28, 2023

Uh oh!

hadoop-yetus commented Jun 28, 2023

Uh oh!

xinglin commented Jun 28, 2023

Uh oh!

hadoop-yetus commented Jun 28, 2023

Uh oh!

hadoop-yetus commented Jun 28, 2023

Uh oh!

xinglin commented Jun 29, 2023

Uh oh!

xinglin commented Jul 3, 2023

Uh oh!

Uh oh!

HDFS-17055 Export HAState as a metric from Namenode for monitoring #5790

HDFS-17055 Export HAState as a metric from Namenode for monitoring #5790

Uh oh!

Conversation

xinglin commented Jun 27, 2023

Description of PR

How was this patch tested?

Uh oh!

hadoop-yetus commented Jun 28, 2023

Uh oh!

hadoop-yetus commented Jun 28, 2023

Uh oh!

xinglin commented Jun 28, 2023

Uh oh!

hadoop-yetus commented Jun 28, 2023

Uh oh!

hadoop-yetus commented Jun 28, 2023

Uh oh!

xinglin commented Jun 29, 2023

Uh oh!

xinglin commented Jul 3, 2023

Uh oh!

Uh oh!