Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDDS-1810. SCM command to Activate and Deactivate pipelines. #1224

Closed
wants to merge 1 commit into from

Conversation

nandakumar131
Copy link
Contributor

It will be useful to have scm command to temporarily deactivate and re-activate a pipeline. This will help us a lot in debugging a pipeline.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 76 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
0 mvndep 84 Maven dependency ordering for branch
+1 mvninstall 617 trunk passed
+1 compile 367 trunk passed
+1 checkstyle 71 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 928 branch has no errors when building and testing our client artifacts.
+1 javadoc 164 trunk passed
0 spotbugs 430 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 630 trunk passed
_ Patch Compile Tests _
0 mvndep 30 Maven dependency ordering for patch
+1 mvninstall 560 the patch passed
+1 compile 369 the patch passed
+1 cc 369 the patch passed
+1 javac 369 the patch passed
+1 checkstyle 76 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 740 patch has no errors when building and testing our client artifacts.
+1 javadoc 164 the patch passed
+1 findbugs 647 the patch passed
_ Other Tests _
+1 unit 348 hadoop-hdds in the patch passed.
-1 unit 2120 hadoop-ozone in the patch failed.
+1 asflicense 43 The patch does not generate ASF License warnings.
8203
Reason Tests
Failed junit tests hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
hadoop.ozone.om.TestScmSafeMode
hadoop.hdds.scm.pipeline.TestPipelineClose
hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures
hadoop.ozone.client.rpc.TestOzoneRpcClient
hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory
hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/1/artifact/out/Dockerfile
GITHUB PR #1224
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc
uname Linux 9ca97659c82a 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 065cbc6
Default Java 1.8.0_222
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/1/testReport/
Max. process+thread count 4698 (vs. ulimit of 5500)
modules C: hadoop-hdds/client hadoop-hdds/common hadoop-hdds/server-scm hadoop-hdds/tools hadoop-ozone/integration-test U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/1/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@xiaoyuyao
Copy link
Contributor

Will deactive trigger close containers on the pipeline? what is the difference between close and recreate?

@nandakumar131
Copy link
Contributor Author

nandakumar131 commented Aug 5, 2019

Will deactive trigger close containers on the pipeline?

No, we will not close the containers. We will not do anything to the pipeline other than moving it out of allocation path.

what is the difference between close and recreate?

Activate and de-activate is used mainly for debugging purpose, we don't want to create a new pipeline here.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 91 Docker mode activated.
_ Prechecks _
+1 dupname 1 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
0 mvndep 81 Maven dependency ordering for branch
+1 mvninstall 637 trunk passed
+1 compile 385 trunk passed
+1 checkstyle 73 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 936 branch has no errors when building and testing our client artifacts.
+1 javadoc 179 trunk passed
0 spotbugs 476 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 693 trunk passed
_ Patch Compile Tests _
0 mvndep 78 Maven dependency ordering for patch
+1 mvninstall 569 the patch passed
+1 compile 383 the patch passed
+1 cc 383 the patch passed
+1 javac 383 the patch passed
+1 checkstyle 80 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 789 patch has no errors when building and testing our client artifacts.
+1 javadoc 193 the patch passed
+1 findbugs 740 the patch passed
_ Other Tests _
-1 unit 258 hadoop-hdds in the patch failed.
-1 unit 2277 hadoop-ozone in the patch failed.
+1 asflicense 43 The patch does not generate ASF License warnings.
8648
Reason Tests
Failed junit tests hadoop.ozone.container.ozoneimpl.TestOzoneContainer
hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
hadoop.ozone.om.TestScmSafeMode
hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures
hadoop.ozone.client.rpc.TestOzoneRpcClient
hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory
hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures
hadoop.ozone.client.rpc.TestWatchForCommit
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/2/artifact/out/Dockerfile
GITHUB PR #1224
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc
uname Linux 7499578d1bdd 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 3cc0ace
Default Java 1.8.0_222
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/2/artifact/out/patch-unit-hadoop-hdds.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/2/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/2/testReport/
Max. process+thread count 4541 (vs. ulimit of 5500)
modules C: hadoop-hdds/client hadoop-hdds/common hadoop-hdds/server-scm hadoop-hdds/tools hadoop-ozone/integration-test U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/2/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 45 Docker mode activated.
_ Prechecks _
+1 dupname 1 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
0 mvndep 25 Maven dependency ordering for branch
+1 mvninstall 607 trunk passed
+1 compile 383 trunk passed
+1 checkstyle 73 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 853 branch has no errors when building and testing our client artifacts.
+1 javadoc 157 trunk passed
0 spotbugs 453 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 660 trunk passed
_ Patch Compile Tests _
0 mvndep 31 Maven dependency ordering for patch
+1 mvninstall 592 the patch passed
+1 compile 381 the patch passed
+1 cc 381 the patch passed
+1 javac 381 the patch passed
+1 checkstyle 72 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 650 patch has no errors when building and testing our client artifacts.
+1 javadoc 155 the patch passed
-1 findbugs 363 hadoop-ozone in the patch failed.
_ Other Tests _
+1 unit 284 hadoop-hdds in the patch passed.
-1 unit 311 hadoop-ozone in the patch failed.
+1 asflicense 36 The patch does not generate ASF License warnings.
5995
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/3/artifact/out/Dockerfile
GITHUB PR #1224
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc
uname Linux fb4728d419e8 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 00b5a27
Default Java 1.8.0_212
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/3/artifact/out/patch-findbugs-hadoop-ozone.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/3/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/3/testReport/
Max. process+thread count 1297 (vs. ulimit of 5500)
modules C: hadoop-hdds/client hadoop-hdds/common hadoop-hdds/server-scm hadoop-hdds/tools hadoop-ozone/integration-test U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/3/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 0 Docker mode activated.
-1 patch 13 #1224 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.
Subsystem Report/Notes
GITHUB PR #1224
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/4/console
versions git=2.7.4
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 0 Docker mode activated.
-1 patch 15 #1224 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.
Subsystem Report/Notes
GITHUB PR #1224
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/5/console
versions git=2.7.4
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 0 Docker mode activated.
-1 patch 12 #1224 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.
Subsystem Report/Notes
GITHUB PR #1224
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/6/console
versions git=2.7.4
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 152 Docker mode activated.
_ Prechecks _
+1 dupname 1 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
0 mvndep 54 Maven dependency ordering for branch
+1 mvninstall 853 trunk passed
+1 compile 433 trunk passed
+1 checkstyle 89 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 1088 branch has no errors when building and testing our client artifacts.
+1 javadoc 199 trunk passed
0 spotbugs 519 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 754 trunk passed
_ Patch Compile Tests _
0 mvndep 37 Maven dependency ordering for patch
+1 mvninstall 688 the patch passed
+1 compile 437 the patch passed
+1 cc 437 the patch passed
+1 javac 437 the patch passed
+1 checkstyle 90 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 833 patch has no errors when building and testing our client artifacts.
+1 javadoc 211 the patch passed
+1 findbugs 789 the patch passed
_ Other Tests _
+1 unit 374 hadoop-hdds in the patch passed.
-1 unit 2346 hadoop-ozone in the patch failed.
+1 asflicense 51 The patch does not generate ASF License warnings.
9680
Reason Tests
Failed junit tests hadoop.ozone.om.TestOzoneManager
hadoop.ozone.container.server.TestSecureContainerServer
hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/7/artifact/out/Dockerfile
GITHUB PR #1224
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc
uname Linux 53efbb2436d6 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 20064b6
Default Java 1.8.0_212
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/7/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/7/testReport/
Max. process+thread count 5262 (vs. ulimit of 5500)
modules C: hadoop-hdds/client hadoop-hdds/common hadoop-hdds/server-scm hadoop-hdds/tools hadoop-ozone/integration-test U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/7/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 42 Docker mode activated.
_ Prechecks _
+1 dupname 1 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
0 mvndep 27 Maven dependency ordering for branch
+1 mvninstall 592 trunk passed
+1 compile 381 trunk passed
+1 checkstyle 81 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 870 branch has no errors when building and testing our client artifacts.
+1 javadoc 179 trunk passed
0 spotbugs 439 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 638 trunk passed
_ Patch Compile Tests _
0 mvndep 35 Maven dependency ordering for patch
+1 mvninstall 558 the patch passed
-1 compile 259 hadoop-ozone in the patch failed.
-1 cc 259 hadoop-ozone in the patch failed.
-1 javac 259 hadoop-ozone in the patch failed.
+1 checkstyle 92 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 668 patch has no errors when building and testing our client artifacts.
+1 javadoc 195 the patch passed
-1 findbugs 394 hadoop-ozone in the patch failed.
_ Other Tests _
+1 unit 307 hadoop-hdds in the patch passed.
-1 unit 356 hadoop-ozone in the patch failed.
+1 asflicense 41 The patch does not generate ASF License warnings.
6215
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/8/artifact/out/Dockerfile
GITHUB PR #1224
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc
uname Linux 9a00fc4e8011 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 3329257
Default Java 1.8.0_222
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/8/artifact/out/patch-compile-hadoop-ozone.txt
cc https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/8/artifact/out/patch-compile-hadoop-ozone.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/8/artifact/out/patch-compile-hadoop-ozone.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/8/artifact/out/patch-findbugs-hadoop-ozone.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/8/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/8/testReport/
Max. process+thread count 1314 (vs. ulimit of 5500)
modules C: hadoop-hdds/client hadoop-hdds/common hadoop-hdds/server-scm hadoop-hdds/tools hadoop-ozone/integration-test U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/8/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@bshashikant
Copy link
Contributor

Thanks @nandakumar131 for working on this. The changes look good to me and the feature is good too have for debugging purpose. I am +1 on the change.

Copy link
Member

@elek elek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 after fixing the small typo.

RpcController controller, DeactivatePipelineRequestProto request)
throws ServiceException {
try (Scope ignored = TracingUtil
.importAndCreateScope("activatePipeline", request.getTraceID())) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
.importAndCreateScope("activatePipeline", request.getTraceID())) {
.importAndCreateScope("deactivatePipeline", request.getTraceID())) {

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 38 Docker mode activated.
_ Prechecks _
+1 dupname 1 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
0 mvndep 59 Maven dependency ordering for branch
+1 mvninstall 663 trunk passed
+1 compile 434 trunk passed
+1 checkstyle 85 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 1023 branch has no errors when building and testing our client artifacts.
+1 javadoc 212 trunk passed
0 spotbugs 503 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 730 trunk passed
_ Patch Compile Tests _
0 mvndep 38 Maven dependency ordering for patch
+1 mvninstall 635 the patch passed
+1 compile 448 the patch passed
+1 cc 448 the patch passed
+1 javac 448 the patch passed
+1 checkstyle 88 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 822 patch has no errors when building and testing our client artifacts.
+1 javadoc 192 the patch passed
+1 findbugs 667 the patch passed
_ Other Tests _
+1 unit 305 hadoop-hdds in the patch passed.
-1 unit 1705 hadoop-ozone in the patch failed.
+1 asflicense 50 The patch does not generate ASF License warnings.
8367
Reason Tests
Failed junit tests hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
hadoop.ozone.om.TestSecureOzoneManager
hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion
hadoop.ozone.client.rpc.TestFailureHandlingByClient
hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient
hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/9/artifact/out/Dockerfile
GITHUB PR #1224
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc
uname Linux fa4556860f0c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 915cbc9
Default Java 1.8.0_222
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/9/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/9/testReport/
Max. process+thread count 5324 (vs. ulimit of 5500)
modules C: hadoop-hdds/client hadoop-hdds/common hadoop-hdds/server-scm hadoop-hdds/tools hadoop-ozone/integration-test U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1224/9/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@nandakumar131
Copy link
Contributor Author

Thanks @elek for the review, I will fix the typo while committing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants