Handle shard ownership lost when reading history tasks #7101
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alfred-landrum
approved these changes
Jan 17, 2025
pdoerner
approved these changes
Jan 17, 2025
ychebotarev
approved these changes
Jan 17, 2025
| ctx context.Context, | ||
| request *persistence.GetHistoryTasksRequest, | ||
| ) (*persistence.GetHistoryTasksResponse, error) { | ||
| if err := s.errorByState(); err != nil { |
There was a problem hiding this comment.
in I understand correctly main change - you "wrap" s.executionManager.GetHistoryTask with
s.errorByState()
and
s.handleReadError()
?
There was a problem hiding this comment.
Yes. s.handleReadError is the key part which will check for shardOwnership lost and will put shard context in an invalid state. Then a new shard will be created once shard controller realizes the shard context is invalid.
yiminc
approved these changes
Jan 17, 2025
stephanos
pushed a commit
to stephanos/temporal
that referenced
this pull request
Jan 17, 2025
## What changed? <!-- Describe what has changed in this PR --> - Wrap ExecutionManager.GetHistoryTasks in shard context to handler shardOwnershipLost errors. - Will need to follow up and audit all usage of this method and other methods on execution manager. This PR is only for fixing the issue we are seeing. ## Why? <!-- Tell your future self why have you made these changes --> - If shard has no (api) traffic, and ownership already got lost in the background (from some persistence implementations). Task processing can get stuck forever trying to load tasks and see this shard ownership lost error. ## How did you test it? <!-- How have you verified this change? Tested locally? Added a unit test? Checked in staging env? --> - Unit test ## Potential risks <!-- Assuming the worst case, what can be broken when deploying this change to production? --> ## Documentation <!-- Have you made sure this change doesn't falsify anything currently stated in `docs/`? If significant new behavior is added, have you described that in `docs/`? --> ## Is hotfix candidate? <!-- Is this PR a hotfix candidate or does it require a notification to be sent to the broader community? (Yes/No) -->
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changed?
Why?
How did you test it?
Potential risks
Documentation
Is hotfix candidate?