Repository navigation
fix(core): reset FAILED JOIN tasks to IN_PROGRESS during retry walk-up - #1274
Open
nthmost-orkes wants to merge 7 commits into
Open
nthmost-orkes wants to merge 7 commits into
nthmost-orkes wants to merge 7 commits into
Conversation
In updateAndPushParents, FAILED and TIMED_OUT JOIN tasks (e.g. inner_join, top_join) were silently skipped during the sibling-processing loop. The existing adjustStateIfSubWorkflowChanged mechanism tries to reset them via resetUnsuccessfulJoinTasks, but it consumes exactly one subworkflowChanged flag per decide() call. Under load (redis-es8 CI), decide() can run before all flags are consumed, finding a FAILED built-in JOIN and throwing TerminateWorkflowException — collapsing the parent workflow that should be resuming after the child retry. Fix: add an else-if branch for FAILED/TIMED_OUT JOIN tasks in the loop, resetting status to IN_PROGRESS and queuing them immediately. This matches what resetUnsuccessfulJoinTasks does but happens unconditionally during the walk-up, removing the race with subworkflowChanged flag consumption. Fixes testRetryNestedSubWorkflowsReschedulesSiblingsAtEveryLevel (e2e redis-es8).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
updateAndPushParents, FAILED and TIMED_OUT JOIN tasks were silently skipped in the sibling-processing loop.subworkflowChanged-flag race in CI alloweddecide()to run AFTER the sibling retry but BEFORE the flag was consumed, finding a FAILED built-in JOIN and throwingTerminateWorkflowException— collapsing the parent workflow that should have been resuming.else ifbranch resetting FAILED/TIMED_OUT JOIN tasks toIN_PROGRESSand queuing them immediately duringupdateAndPushParents, unconditionally, so no subsequentdecide()can encounter a FAILED built-in JOIN during the retry walk-up.Root cause
updateAndPushParentsloops over the parent's tasks and:But it had no handling for FAILED or TIMED_OUT JOIN tasks. They remained FAILED in Redis after the walk-up completed.
adjustStateIfSubWorkflowChangedis supposed to clean them up viaresetUnsuccessfulJoinTasks, but that mechanism consumes exactly onesubworkflowChangedflag perdecide()call. Under fast execution (redis-es8 CI), adecide()call can arrive for the parent workflow while some flags are still unconsumed and findinner_join = FAILED—DeciderService.retry()sees a FAILED built-in task and throwsTerminateWorkflowException, terminating the parent.Fix
Added an
else if ((task.getStatus() == FAILED || task.getStatus() == TIMED_OUT) && TASK_TYPE_JOIN.equals(task.getTaskType()))branch inupdateAndPushParents's sibling loop that resets the JOIN toIN_PROGRESSand callsaddTaskToQueue, identical to whatresetUnsuccessfulJoinTasksdoes.User impact
Retrying a nested sub-workflow (e.g.
workflowClient.retryWorkflow(...)) could cause parent workflows to become TERMINATED instead of RUNNING when running against a Redis/ES8 backend, because FAILED JOIN tasks upstream were not cleaned up before the next decide cycle ran.Orkes parity
No Orkes-parallel file — this is a bug fix to an existing method in
WorkflowExecutorOps.java.