HDDS-2107. Datanodes should retry forever to connect to SCM in an… #1424
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ecure environment
|
/label ozone |
|
@xiaoyuyao @hanishakoneru @anuengineer @elek @bharatviswa504 Please review |
|
💔 -1 overall
This message was automatically generated. |
adoroszlai
reviewed
Sep 11, 2019
There was a problem hiding this comment.
Hi @vivekratnavel,
As far as I see, DataNode already tries forever due to the main loop in the state machine:
You can verify this by starting DataNode without SCM, and setting the IP for scm to the DataNode's own address:
cd hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozone
docker-compose up -d datanode
docker-compose exec datanode bash -c "tail -1 /etc/hosts | sed 's/\t\+[a-z0-9]*$/ scm/' | sudo tee -a /etc/hosts"
docker-compose logs -f --tail=10 datanode
Result:
...
datanode_1 | 2019-09-11 12:29:39 INFO Client:948 - Retrying connect to server: scm/192.168.0.2:9861. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
datanode_1 | 2019-09-11 12:29:39 ERROR EndpointStateMachine:204 - Unable to communicate to SCM server at scm:9861 for past 300 seconds.
...
datanode_1 | 2019-09-11 12:29:40 INFO Client:948 - Retrying connect to server: scm/192.168.0.2:9861. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
...
|
@adoroszlai You are right. With this change, we don't get the error from |
|
Thank you @vivekratnavel for working on this. |
|
@hanishakoneru Sure. |
|
Thank you @vivekratnavel. +1. I will commit it. |
amahussein
pushed a commit
to amahussein/hadoop
that referenced
this pull request
Oct 29, 2019
…ecure environment (apache#1424)
RogPodge
pushed a commit
to RogPodge/hadoop
that referenced
this pull request
Mar 25, 2020
…ecure environment (apache#1424)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
… unsecure environment
In an unsecure environment, the datanodes try upto 10 times after waiting for 1000 milliseconds each time before throwing this error:
This PR fixes that issue by having datanodes try forever to connect with SCM and not throw an error from the state machine.
I have also increased timeouts on a unit test to improve its stability.