Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Introduce grace period for failed connections in the tree service #1456

Closed
Tracked by #1322
fyrchik opened this issue May 30, 2022 · 2 comments · Fixed by #1488
Closed
Tracked by #1322

Introduce grace period for failed connections in the tree service #1456

fyrchik opened this issue May 30, 2022 · 2 comments · Fixed by #1488
Assignees
@fyrchik
Labels
U3 Regular

Comments

@fyrchik
Copy link
Contributor

fyrchik commented May 30, 2022 •
edited
Loading

If a connection to the node fails, we should wait some time before trying to connect again, at least in the replicator.

Originally caught at #1445:

  1. Connect to the TLS node in dev-env with grpc.Insecure
  2. Fail with timeout.
  3. Latency of the client operations is increased to 2 seconds (dial timeout) because all workers are busy with connecting to the failed node and channels eventually become full.

Another solution would be to have a dedicated goroutine for each client, but this requires some refactoring and is somewhat related to #1447 .
@fyrchik fyrchik mentioned this issue May 30, 2022
Open
23 tasks
@fyrchik fyrchik mentioned this issue Jun 7, 2022
Merged
@carpawell
Copy link
Member

was it solved in the #1488? or we do not close the issue until the feature branch is merged?

@fyrchik
Copy link
Contributor Author

fyrchik commented Jun 21, 2022

Closed via #1488

@alexchetaev alexchetaev added the U3 Regular label Sep 1, 2022
aprasolova pushed a commit to aprasolova/neofs-node that referenced this issue Oct 19, 2022
@fyrchik @aprasolova
[nspcc-dev#1456] services/tree: Wait some time before reconnecting af…
9988645 
…ter failure

In case node is down or failing for some reason, we can expect `Dial` to
fail. In case we actively try to replicate and `Dial` always takes 2
seconds, replication-related channels quickly become full. That affects
latency of all other write operations.

Signed-off-by: Evgenii Stratonikov <evgeniy@nspcc.ru>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fyrchik fyrchik

Labels
U3 Regular
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[#1456] services/tree: wait some time before reconnecting after failure fyrchik/neofs-node
3 participants
@fyrchik @carpawell @alexchetaev