Skip to content

Make recovery source partially non-blocking #37291

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 14 commits into from
Jan 12, 2019
Merged

Make recovery source partially non-blocking #37291

merged 14 commits into from
Jan 12, 2019

Conversation

dnhatn
Copy link
Member

@dnhatn dnhatn commented Jan 10, 2019

Today a peer-recovery may run into a deadlock if the value of
node_concurrent_recoveries is too high. This happens because the
peer-recovery is executed in a blocking fashion. This commit attempts to
make the recovery source partially non-blocking. I will make three
follow-ups to make it fully non-blocking: (1) send translog operations,
(2) primary relocation, (3) send commit files.

Relates #36195

@dnhatn
Partially make recovery source non-blocking
627d6d5 
Today a peer-recovery may run into a deadlock if the value of
`node_concurrent_recoveries` is too high. This happens because the
peer-recovery is executed in a blocking fashion. This commit attempts to
make the recovery source partially non-blocking. I will make three
follow-ups to make it fully non-blocking: (1) send translog operations,
(2) primary relocation, (3) send commit files.
@dnhatn dnhatn added >enhancement :Distributed Indexing/Recovery Anything around constructing a new shard, either from a local or a remote source. v7.0.0 v6.7.0 labels Jan 10, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@dnhatn dnhatn changed the title Partially make recovery source non-blocking Make recovery source partially non-blocking Jan 10, 2019
s1monw
s1monw suggested changes Jan 10, 2019
View reviewed changes
Copy link
Contributor

@s1monw s1monw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did a first pass. I would love to minimize the steps we make async in this PR even further.

This was referenced Jan 10, 2019
Merged
Merged
@dnhatn
Copy link
Member Author

dnhatn commented Jan 10, 2019

@s1monw Thanks for looking. I've minimized changes in this PR - just try to provide the infra for the next steps. Would you please take another look?

@dnhatn dnhatn requested a review from s1monw January 10, 2019 23:26
@dnhatn
Copy link
Member Author

dnhatn commented Jan 11, 2019

@elasticmachine run gradle build tests 1

dnhatn added a commit that referenced this pull request Jan 11, 2019
@dnhatn
Add runAfter and notifyOnce wrapper to ActionListener (#37331)
360c430 
Relates #37291
s1monw
s1monw suggested changes Jan 11, 2019
View reviewed changes
Copy link
Contributor

@s1monw s1monw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

took another round

@dnhatn
Copy link
Member Author

dnhatn commented Jan 11, 2019

@s1monw I pushed changes. Can you have another look?

@dnhatn dnhatn requested a review from s1monw January 11, 2019 17:59
dnhatn added a commit that referenced this pull request Jan 11, 2019
@dnhatn
Introduce StepListener (#37327)
70cee18 
This commit introduces StepListener which provides a simple way to write
a flow consisting of multiple asynchronous steps without having nested
callbacks.

Relates #37291
dnhatn added a commit that referenced this pull request Jan 12, 2019
@dnhatn
Add runAfter and notifyOnce wrapper to ActionListener (#37331)
773d4c2 
Relates #37291
dnhatn added a commit that referenced this pull request Jan 12, 2019
@dnhatn
Introduce StepListener (#37327)
3e419ba 
This commit introduces StepListener which provides a simple way to write
a flow consisting of multiple asynchronous steps without having nested
callbacks.

Relates #37291
s1monw
s1monw approved these changes Jan 12, 2019
View reviewed changes
Copy link
Contributor

@s1monw s1monw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM left 2 comments

@dnhatn
Copy link
Member Author

dnhatn commented Jan 12, 2019

Thanks @s1monw.

@dnhatn dnhatn merged commit 44a1071 into elastic:master Jan 12, 2019
@dnhatn dnhatn deleted the non-blocking-recovery branch January 12, 2019 17:49
dnhatn added a commit to dnhatn/elasticsearch that referenced this pull request Jan 12, 2019
@dnhatn
Make finalize step of recovery source non-blocking
97e9196 
Relates elastic#37291
@dnhatn dnhatn mentioned this pull request Jan 12, 2019
Merged
dnhatn added a commit that referenced this pull request Jan 13, 2019
@dnhatn
Make recovery source partially non-blocking (#37291)
32f5f9f 
Today a peer-recovery may run into a deadlock if the value of
node_concurrent_recoveries is too high. This happens because the
peer-recovery is executed in a blocking fashion. This commit attempts
to make the recovery source partially non-blocking. I will make three
follow-ups to make it fully non-blocking: (1) send translog operations,
(2) primary relocation, (3) send commit files.

Relates #36195
dnhatn added a commit that referenced this pull request Jan 14, 2019
@dnhatn
Make finalize step of recovery source non-blocking (#37388)
397f315 
Relates #37291
@dnhatn dnhatn mentioned this pull request Jan 15, 2019
Merged
dnhatn added a commit that referenced this pull request Jan 15, 2019
@dnhatn
Make finalize step of recovery source non-blocking (#37388)
821bb24 
Relates #37291
dnhatn added a commit that referenced this pull request Jan 15, 2019
@dnhatn
Prepare to make send translog of recovery non-blocking (#37458)
6647122 
This commit prepares the required infra to make send a translog snapshot
of the recovery source non-blocking. I'll make a follow-up to make the send
snapshot method non-blocking.

Relates #37291
dnhatn added a commit that referenced this pull request Jan 23, 2019
@dnhatn
Prepare to make send translog of recovery non-blocking (#37458)
59f9784 
This commit prepares the required infra to make send a translog snapshot
of the recovery source non-blocking. I'll make a follow-up to make the send
snapshot method non-blocking.

Relates #37291
kovrus added a commit to crate/crate that referenced this pull request Sep 11, 2019
@kovrus
Make recovery source partially non-blocking.
340a4e8 
Backport of elastic/elasticsearch#37291
kovrus added a commit to crate/crate that referenced this pull request Sep 11, 2019
@kovrus
Make recovery source partially non-blocking.
41d2f85 
Backport of elastic/elasticsearch#37291
kovrus added a commit to crate/crate that referenced this pull request Sep 12, 2019
@kovrus
Make recovery source partially non-blocking.
db318d0 
Backport of elastic/elasticsearch#37291
kovrus added a commit to crate/crate that referenced this pull request Sep 12, 2019
@kovrus
Make recovery source partially non-blocking.
8e2a0d5 
Backport of elastic/elasticsearch#37291
kovrus added a commit to crate/crate that referenced this pull request Sep 12, 2019
@kovrus
Make recovery source partially non-blocking.
23a2b07 
Backport of elastic/elasticsearch#37291
mergify bot pushed a commit to crate/crate that referenced this pull request Sep 17, 2019
@kovrus @mergify
Make recovery source partially non-blocking.
4e2185e 
Backport of elastic/elasticsearch#37291
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@s1monw s1monw s1monw approved these changes

@ywelsch ywelsch Awaiting requested review from ywelsch

@DaveCTurner DaveCTurner Awaiting requested review from DaveCTurner

Assignees
No one assigned
Labels
:Distributed Indexing/Recovery Anything around constructing a new shard, either from a local or a remote source. >enhancement v6.7.0 v7.0.0-beta1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@dnhatn @elasticmachine @s1monw @colings86