HDDS-1136 : Add metric counters to capture the RocksDB checkpointing statistics. #537

avijayanhwx · 2019-03-01T03:34:07Z

Added metric gauges for tracking DB checkpointing statistics. The OMMetrics class will hold these guages at any instant. These can be pulled from OM by Recon.

Testing done
Integration test for Servlet method that gets the OM DB checkpoint added.
Manually verified the patch on single node Ozone cluster.

…statistics.

hadoop-yetus · 2019-03-01T05:31:11Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	25	Docker mode activated.
		_ Prechecks _
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	23	Maven dependency ordering for branch
+1	mvninstall	974	trunk passed
+1	compile	991	trunk passed
+1	checkstyle	199	trunk passed
+1	mvnsite	131	trunk passed
+1	shadedclient	1024	branch has no errors when building and testing our client artifacts.
0	findbugs	0	Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1	findbugs	124	trunk passed
+1	javadoc	92	trunk passed
		_ Patch Compile Tests _
0	mvndep	21	Maven dependency ordering for patch
-1	mvninstall	23	integration-test in the patch failed.
+1	compile	991	the patch passed
+1	javac	991	the patch passed
+1	checkstyle	183	the patch passed
+1	mvnsite	110	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	667	patch has no errors when building and testing our client artifacts.
0	findbugs	0	Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1	findbugs	141	the patch passed
-1	javadoc	23	common in the patch failed.
		_ Other Tests _
-1	unit	81	common in the patch failed.
-1	unit	1012	integration-test in the patch failed.
+1	unit	47	ozone-manager in the patch passed.
+1	asflicense	36	The patch does not generate ASF License warnings.
		6949

Reason	Tests
Failed junit tests	hadoop.hdds.security.x509.certificate.client.TestDefaultCertificateClient
	hadoop.ozone.scm.pipeline.TestPipelineManagerMXBean
	hadoop.ozone.ozShell.TestOzoneShell
	hadoop.ozone.om.TestOzoneManager
	hadoop.ozone.scm.TestSCMNodeManagerMXBean
	hadoop.ozone.scm.node.TestQueryNode
	hadoop.ozone.scm.TestXceiverClientMetrics

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-537/1/artifact/out/Dockerfile
GITHUB PR	#537
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 7cfa2d420110 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `eae3db9`
maven	version: Apache Maven 3.3.9
Default Java	1.8.0_191
findbugs	v3.1.0-RC1
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-537/1/artifact/out/patch-mvninstall-hadoop-ozone_integration-test.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-537/1/artifact/out/patch-javadoc-hadoop-hdds_common.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-537/1/artifact/out/patch-unit-hadoop-hdds_common.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-537/1/artifact/out/patch-unit-hadoop-ozone_integration-test.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-537/1/testReport/
Max. process+thread count	3239 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-537/1/console
Powered by	Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

…statistics.(2)

hadoop-yetus · 2019-03-01T15:38:30Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	29	Docker mode activated.
		_ Prechecks _
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	72	Maven dependency ordering for branch
+1	mvninstall	1067	trunk passed
+1	compile	1018	trunk passed
+1	checkstyle	195	trunk passed
-1	mvnsite	56	integration-test in trunk failed.
-1	mvnsite	31	ozone-manager in trunk failed.
+1	shadedclient	1039	branch has no errors when building and testing our client artifacts.
0	findbugs	0	Skipped patched modules with no Java source: hadoop-ozone/integration-test
-1	findbugs	27	ozone-manager in trunk failed.
+1	javadoc	90	trunk passed
		_ Patch Compile Tests _
0	mvndep	23	Maven dependency ordering for patch
-1	mvninstall	19	integration-test in the patch failed.
-1	mvninstall	18	ozone-manager in the patch failed.
+1	compile	988	the patch passed
+1	javac	988	the patch passed
+1	checkstyle	203	the patch passed
-1	mvnsite	31	integration-test in the patch failed.
-1	mvnsite	28	ozone-manager in the patch failed.
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	651	patch has no errors when building and testing our client artifacts.
0	findbugs	0	Skipped patched modules with no Java source: hadoop-ozone/integration-test
-1	findbugs	27	ozone-manager in the patch failed.
+1	javadoc	96	the patch passed
		_ Other Tests _
-1	unit	80	common in the patch failed.
-1	unit	28	integration-test in the patch failed.
-1	unit	26	ozone-manager in the patch failed.
+1	asflicense	34	The patch does not generate ASF License warnings.
		6081

Reason	Tests
Failed junit tests	hadoop.hdds.security.x509.certificate.client.TestDefaultCertificateClient

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-537/2/artifact/out/Dockerfile
GITHUB PR	#537
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 4a88d93d9cad 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / dcaca19
maven	version: Apache Maven 3.3.9
Default Java	1.8.0_191
mvnsite	https://builds.apache.org/job/hadoop-multibranch/job/PR-537/2/artifact/out/branch-mvnsite-hadoop-ozone_integration-test.txt
mvnsite	https://builds.apache.org/job/hadoop-multibranch/job/PR-537/2/artifact/out/branch-mvnsite-hadoop-ozone_ozone-manager.txt
findbugs	v3.1.0-RC1
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-537/2/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-537/2/artifact/out/patch-mvninstall-hadoop-ozone_integration-test.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-537/2/artifact/out/patch-mvninstall-hadoop-ozone_ozone-manager.txt
mvnsite	https://builds.apache.org/job/hadoop-multibranch/job/PR-537/2/artifact/out/patch-mvnsite-hadoop-ozone_integration-test.txt
mvnsite	https://builds.apache.org/job/hadoop-multibranch/job/PR-537/2/artifact/out/patch-mvnsite-hadoop-ozone_ozone-manager.txt
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-537/2/artifact/out/patch-findbugs-hadoop-ozone_ozone-manager.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-537/2/artifact/out/patch-unit-hadoop-hdds_common.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-537/2/artifact/out/patch-unit-hadoop-ozone_integration-test.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-537/2/artifact/out/patch-unit-hadoop-ozone_ozone-manager.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-537/2/testReport/
Max. process+thread count	445 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-537/2/console
Powered by	Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

avijayanhwx · 2019-03-04T23:16:38Z

Patch committed to trunk.

…nRunner tests. Currently all the tests in TestZkLocalApplicationRunner takes around 5 minutes to finish. Reducing the coordination timeout to reduce the test time. Changes in TestZkLocalApplicationRunner test timeout values: * Change debounce timeout from 20 seconds to 2 seconds. * Change task.shutdown timeout from 30 seconds to 2 seconds. * Change barrier timeout from 40 seconds to 2 seconds. After this change, execution time of TestZkLocalApplicationRunner tests has reduced from `310` seconds to `55` seconds. Author: Shanthoosh Venkataraman <svenkataraman@linkedin.com> Reviewers: Jagadish <jagadish@apache.org> Closes apache#537 from shanthoosh/reduce_zk_localAppRunnerTestTime

HDDS-1136 : Add metric counters to capture the RocksDB checkpointing …

ac3436e

…statistics.

elek added the ozone label Mar 1, 2019

HDDS-1136 : Add metric counters to capture the RocksDB checkpointing …

8448b49

…statistics.(2)

avijayanhwx closed this Mar 4, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-1136 : Add metric counters to capture the RocksDB checkpointing statistics. #537

HDDS-1136 : Add metric counters to capture the RocksDB checkpointing statistics. #537

avijayanhwx commented Mar 1, 2019

hadoop-yetus commented Mar 1, 2019

hadoop-yetus commented Mar 1, 2019

avijayanhwx commented Mar 4, 2019

HDDS-1136 : Add metric counters to capture the RocksDB checkpointing statistics. #537

HDDS-1136 : Add metric counters to capture the RocksDB checkpointing statistics. #537

Conversation

avijayanhwx commented Mar 1, 2019

hadoop-yetus commented Mar 1, 2019

hadoop-yetus commented Mar 1, 2019

avijayanhwx commented Mar 4, 2019