Skip to content

HDDS-1353 : Metrics scm_pipeline_metrics_num_pipeline_creation_failed keeps increasing because of BackgroundPipelineCreator. #681

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Apr 4, 2019
Merged

HDDS-1353 : Metrics scm_pipeline_metrics_num_pipeline_creation_failed keeps increasing because of BackgroundPipelineCreator. #681

merged 3 commits into from
Apr 4, 2019

Conversation

avijayanhwx
Copy link
Contributor

There is a BackgroundPipelineCreator thread in SCM which runs in a fixed interval and tries to create pipelines. This BackgroundPipelineCreator uses IOException as exit criteria (no more pipelines can be created). In each run of BackgroundPipelineCreator we exit when we are not able to create any more pipelines, i.e. when we get IOException while trying to create the pipeline. This means that scm_pipeline_metrics_num_pipeline_creation_failed value will get incremented in each run of BackgroundPipelineCreator.

HDDS-1353 : Metrics scm_pipeline_metrics_num_pipeline_creation_failed…
0d7986b 
… keeps increasing because of BackgroundPipelineCreator.
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 39 Docker mode activated.
_ Prechecks _
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
0 mvndep 94 Maven dependency ordering for branch
+1 mvninstall 1140 trunk passed
+1 compile 1057 trunk passed
+1 checkstyle 213 trunk passed
+1 mvnsite 117 trunk passed
+1 shadedclient 1085 branch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1 findbugs 49 trunk passed
+1 javadoc 56 trunk passed
_ Patch Compile Tests _
0 mvndep 24 Maven dependency ordering for patch
+1 mvninstall 65 the patch passed
+1 compile 1031 the patch passed
+1 javac 1031 the patch passed
+1 checkstyle 209 the patch passed
+1 mvnsite 80 the patch passed
-1 whitespace 0 The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 shadedclient 708 patch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1 findbugs 55 the patch passed
+1 javadoc 54 the patch passed
_ Other Tests _
+1 unit 119 server-scm in the patch passed.
-1 unit 971 integration-test in the patch failed.
+1 asflicense 43 The patch does not generate ASF License warnings.
7130
Reason Tests
Failed junit tests hadoop.ozone.TestMiniOzoneCluster
hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures
hadoop.hdds.scm.pipeline.TestSCMPipelineManager
hadoop.ozone.TestContainerStateMachineIdempotency
hadoop.ozone.om.TestOzoneManagerHA
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-681/1/artifact/out/Dockerfile
GITHUB PR #681
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux b02fcadd3e3e 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct 5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / bfc90bd
maven version: Apache Maven 3.3.9
Default Java 1.8.0_191
findbugs v3.1.0-RC1
whitespace https://builds.apache.org/job/hadoop-multibranch/job/PR-681/1/artifact/out/whitespace-eol.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-681/1/artifact/out/patch-unit-hadoop-ozone_integration-test.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-681/1/testReport/
Max. process+thread count 4138 (vs. ulimit of 5500)
modules C: hadoop-hdds/server-scm hadoop-ozone/integration-test U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-681/1/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

HDDS-1353 : Metrics scm_pipeline_metrics_num_pipeline_creation_failed…
e5f30bc 
… keeps increasing because of BackgroundPipelineCreator.
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 34 Docker mode activated.
_ Prechecks _
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
0 mvndep 66 Maven dependency ordering for branch
+1 mvninstall 1094 trunk passed
+1 compile 1094 trunk passed
+1 checkstyle 203 trunk passed
+1 mvnsite 113 trunk passed
+1 shadedclient 1000 branch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1 findbugs 47 trunk passed
+1 javadoc 49 trunk passed
_ Patch Compile Tests _
0 mvndep 24 Maven dependency ordering for patch
+1 mvninstall 63 the patch passed
+1 compile 920 the patch passed
+1 javac 920 the patch passed
+1 checkstyle 191 the patch passed
+1 mvnsite 79 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 687 patch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1 findbugs 57 the patch passed
+1 javadoc 66 the patch passed
_ Other Tests _
+1 unit 123 server-scm in the patch passed.
-1 unit 678 integration-test in the patch failed.
+1 asflicense 46 The patch does not generate ASF License warnings.
6572
Reason Tests
Failed junit tests hadoop.hdds.scm.pipeline.TestSCMPipelineManager
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-681/2/artifact/out/Dockerfile
GITHUB PR #681
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux d0a668414102 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / cf26811
maven version: Apache Maven 3.3.9
Default Java 1.8.0_191
findbugs v3.1.0-RC1
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-681/2/artifact/out/patch-unit-hadoop-ozone_integration-test.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-681/2/testReport/
Max. process+thread count 4493 (vs. ulimit of 5500)
modules C: hadoop-hdds/server-scm hadoop-ozone/integration-test U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-681/2/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

HDDS-1353 : Metrics scm_pipeline_metrics_num_pipeline_creation_failed…
c6b13f6 
… keeps increasing because of BackgroundPipelineCreator. (Commit 3)
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 43 Docker mode activated.
_ Prechecks _
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
0 mvndep 70 Maven dependency ordering for branch
+1 mvninstall 1182 trunk passed
+1 compile 1148 trunk passed
+1 checkstyle 208 trunk passed
+1 mvnsite 124 trunk passed
+1 shadedclient 1075 branch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1 findbugs 51 trunk passed
+1 javadoc 56 trunk passed
_ Patch Compile Tests _
0 mvndep 61 Maven dependency ordering for patch
+1 mvninstall 87 the patch passed
+1 compile 1085 the patch passed
+1 javac 1085 the patch passed
+1 checkstyle 220 the patch passed
+1 mvnsite 81 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 713 patch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1 findbugs 56 the patch passed
+1 javadoc 56 the patch passed
_ Other Tests _
+1 unit 126 server-scm in the patch passed.
-1 unit 1110 integration-test in the patch failed.
+1 asflicense 55 The patch does not generate ASF License warnings.
7526
Reason Tests
Failed junit tests hadoop.hdds.scm.pipeline.TestRatisPipelineUtils
hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures
hadoop.ozone.TestMiniChaosOzoneCluster
hadoop.ozone.scm.pipeline.TestSCMPipelineMetrics
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-681/3/artifact/out/Dockerfile
GITHUB PR #681
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux b2219f6251b4 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / cf26811
maven version: Apache Maven 3.3.9
Default Java 1.8.0_191
findbugs v3.1.0-RC1
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-681/3/artifact/out/patch-unit-hadoop-ozone_integration-test.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-681/3/testReport/
Max. process+thread count 5249 (vs. ulimit of 5500)
modules C: hadoop-hdds/server-scm hadoop-ozone/integration-test U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-681/3/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@vivekratnavel
Copy link
Contributor

+1 LGTM

nandakumar131
nandakumar131 approved these changes Apr 4, 2019
View reviewed changes
Copy link
Contributor

@nandakumar131 nandakumar131 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM.

@nandakumar131 nandakumar131 merged commit 993f36e into apache:trunk Apr 4, 2019
shanthoosh pushed a commit to shanthoosh/hadoop that referenced this pull request Oct 15, 2019
@jagadish-v0
Reorganize website content, link hyper-links correctly, fix image links
4ae30f1 
Author: Jagadish <jvenkatraman@linkedin.com>

Reviewers: Jagadish<jagadish@apache.org>

Closes apache#681 from vjagadish1989/website-reorg1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hadoop-yetus hadoop-yetus hadoop-yetus left review comments

@nandakumar131 nandakumar131 nandakumar131 approved these changes

Assignees

@avijayanhwx avijayanhwx

Labels
ozone
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@avijayanhwx @hadoop-yetus @vivekratnavel @nandakumar131