Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Scanner] handle current execution to read from blobstore #3435

Merged
merged 10 commits into from
Aug 5, 2020

Conversation

mkolodezny
Copy link
Contributor

No description provided.

yux0 and others added 10 commits August 4, 2020 09:33
…rkflow#3423)

When rangeID changed and sticky task ConditionFailedError happened, it is possible for the sticky tasklist to go into a bad state that none task can be added to it because of endless ConditionFailedError. This will cause latency increase for workflow, because all sticky decision task timeout (and workflow can only progress by using normal tasklist)
Once it happened, the only possible way to recover is for user to restart workers (with those affected tasklists)

We want to stablize the tasklist rangeID so that no need for worker to restart to recover using the sticky tasklist.
This PR use a relatively conservative approach to stop infinite retrying sticky failures on ConditionFailedError. By reducing concurrent access to tasklist, it will be more likely to break the endless ConditionFailedError.

Another approach would be clear stickness on worker side, but there would be bigger changes and risks.
…ow#3434)

* [Scanner] Fixed missing scan type param in activities
@mkolodezny mkolodezny marked this pull request as ready for review August 5, 2020 19:17
@mkolodezny mkolodezny merged commit 6b64e4c into cadence-workflow:master Aug 5, 2020
@coveralls
Copy link

Coverage Status

Coverage decreased (-0.03%) to 66.965% when pulling c5cab82 on mkolodezny:patch into e284980 on uber:master.

mkolodezny added a commit to mkolodezny/cadence that referenced this pull request Aug 20, 2020
yux0 pushed a commit that referenced this pull request Sep 1, 2020
* [Scanner] handle current execution to read from blobstore (#3435)
@mkolodezny mkolodezny deleted the patch branch October 19, 2020 18:37
yux0 pushed a commit to yux0/cadence that referenced this pull request May 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants