HDDS-1569 Support creating multiple pipelines with same datanode #1431
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
timmylicheng
force-pushed
the
hdds-1569
branch
from
9c6f894 to
bbdaf89
Compare
timmylicheng
force-pushed
the
hdds-1569
branch
from
bbdaf89 to
d0a98ff
Compare
|
💔 -1 overall
This message was automatically generated. |
timmylicheng
force-pushed
the
hdds-1569
branch
3 times, most recently
from
a6237e3 to
2300b7e
Compare
|
💔 -1 overall
This message was automatically generated. |
|
/retest |
|
💔 -1 overall
This message was automatically generated. |
ChenSammi
reviewed
Sep 20, 2019
...ds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelinePlacementPolicy.java
Outdated
Show resolved
Hide resolved
timmylicheng
force-pushed
the
hdds-1569
branch
2 times, most recently
from
ae7e4d4 to
cecfd74
Compare
ChenSammi
reviewed
Sep 20, 2019
...erver-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/HealthyPipelineSafeModeRule.java
Outdated
Show resolved
Hide resolved
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestDeadNodeHandler.java
Outdated
Show resolved
Hide resolved
...dds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/safemode/TestSCMSafeModeManager.java
Outdated
Show resolved
Hide resolved
...integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/Test2WayCommitInRatis.java
Outdated
Show resolved
Hide resolved
|
💔 -1 overall
This message was automatically generated. |
timmylicheng
force-pushed
the
hdds-1569
branch
from
cecfd74 to
e9494bd
Compare
|
💔 -1 overall
This message was automatically generated. |
xiaoyuyao
reviewed
Sep 21, 2019
| @@ -71,6 +71,10 @@ public synchronized void addPipeline(Pipeline pipeline) { | |||
| UUID dnId = details.getUuid(); | |||
| dn2ObjectMap.computeIfAbsent(dnId, k -> ConcurrentHashMap.newKeySet()) | |||
| .add(pipeline.getId()); | |||
| dn2ObjectMap.computeIfPresent(dnId, (k, v) -> { | |||
There was a problem hiding this comment.
Line 73 should be able to add(pipeline.getId() if the HashMap exist. Line 74-77 seems redundant.
There was a problem hiding this comment.
From Java doc:https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ConcurrentHashMap.html#computeIfAbsent-K-java.util.function.Function- , it looks like computeIfAbsent is only adding absent member. Line 74 is trying to merge a pipelineId into an existed candidate. This happens when one datanode is assigned to multiple pipelines, which is a classic scenario for multiraft.
xiaoyuyao
reviewed
Sep 21, 2019
...ds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelinePlacementPolicy.java
Outdated
Show resolved
Hide resolved
xiaoyuyao
reviewed
Sep 21, 2019
...op-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
Outdated
Show resolved
Hide resolved
xiaoyuyao
reviewed
Sep 21, 2019
timmylicheng
force-pushed
the
hdds-1569
branch
2 times, most recently
from
0520975 to
1dd4a08
Compare
|
💔 -1 overall
This message was automatically generated. |
timmylicheng
force-pushed
the
hdds-1569
branch
2 times, most recently
from
7a7b665 to
2516f50
Compare
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
timmylicheng
force-pushed
the
hdds-1569
branch
from
2516f50 to
3d60e02
Compare
|
/retest |
|
💔 -1 overall
This message was automatically generated. |
timmylicheng
force-pushed
the
hdds-1569
branch
from
9964f70 to
59a1b96
Compare
|
💔 -1 overall
This message was automatically generated. |
timmylicheng
force-pushed
the
hdds-1569
branch
2 times, most recently
from
4f48790 to
ada4060
Compare
|
💔 -1 overall
This message was automatically generated. |
timmylicheng
force-pushed
the
hdds-1569
branch
from
ada4060 to
937d4d3
Compare
|
💔 -1 overall
This message was automatically generated. |
timmylicheng
force-pushed
the
hdds-1569
branch
from
937d4d3 to
6e4e832
Compare
|
💔 -1 overall
This message was automatically generated. |
timmylicheng
force-pushed
the
hdds-1569
branch
from
6e4e832 to
6d40e53
Compare
|
💔 -1 overall
This message was automatically generated. |
timmylicheng
force-pushed
the
hdds-1569
branch
from
6d40e53 to
eb71f55
Compare
|
💔 -1 overall
This message was automatically generated. |
timmylicheng
force-pushed
the
hdds-1569
branch
from
eb71f55 to
3affb98
Compare
|
💔 -1 overall
This message was automatically generated. |
timmylicheng
force-pushed
the
hdds-1569
branch
2 times, most recently
from
ddb69f6 to
85fb3cb
Compare
|
💔 -1 overall
This message was automatically generated. |
…iton. Also add consideration for CLOSED pipeline to join allocation.
timmylicheng
force-pushed
the
hdds-1569
branch
from
85fb3cb to
a74f841
Compare
|
💔 -1 overall
This message was automatically generated. |
|
Thank you very much to open this pull request. During the weekend the Ozone source code has been moved out from apache/hadoop repository to apache/hadoop-ozone repository. This git commits are rewritten, but the branch of this pull request is also transformed (state of Saturday morning), you can use the new, migrated branch to recreate this pull request. Your pull request is important for us: Can you please re-create your pull request in the new repository? 1. Create a new fork of https://github.com/apache/hadoop-ozone 2. Clone it and have both your fork and the apache repo as remotes: 3. Fetch your migrated branch and push it to your fork. 4. And create the new pull request on the new repository. https://github.com/apache/hadoop-ozone/compare/master...timmylicheng:HDDS-1569?expand=1 If you need more information, please check this wiki page or contact with me (my github user name + apache.org). Thank you, and sorry for the inconvenience. |
Ok I will send out a new PR. How to do mvn build under new repo now? I was not able to do it under hadoop-ozone directory. |
Thank you very much, to take care about the migration of your PRs. (Unfortunately I can't do it with github api as I can't fake the reporter and I would like to keep it) Regarding the build in the new repo. You can do it from the root level of the project:
((
|
CI build failed on new PR: apache/ozone#13. Could you please take a look? |
Sorry, this is may fault. One commit is missing from all of the created pr branches. (restore the README.txt)
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#HDDS-1569 Use PipelinePlacementPolicy to support creating multiple pipelines with same datanode