Fix upstream map index resolution after placeholder expansion #59691
+145
−0
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Fix upstream map index resolution for dynamically mapped task groups when placeholder task instances are replaced during expansion.
After expansion, the upstream placeholder (
map_index = -1) is replaced by the first expanded task instance (map_index = 0). Downstream task instances that are still unexpanded may continue to reference the placeholder, causing incorrect upstream dependency resolution.This change introduces a small helper (
resolve_placeholder_map_index) that is invoked during upstream map index resolution to correctly handle this transition, while preserving existing behavior for already-expanded downstream tasks.Rationale
Placeholder task instances are a temporary pre-expansion representation. Once expansion occurs, they are no longer valid references, but downstream resolution logic may still encounter them. Handling this transition explicitly prevents downstream tasks from treating completed upstream work as unresolved.
This could cause implicit dependencies to be skipped during DAG evaluation under certain trigger rules (such as
NONE_FAILED_MIN_ONE_SUCCESS), particularly when multiple parallel task streams within theMappedTaskGroupconverge on a single downstream task outside the group. As a result, DAG runs could be incorrectly marked as failed or skipped despite valid upstream execution.Please refer to issue #59289 for more context. This PR was opened in response to that. The author of the issue reported the bug in Airflow 3.0.6 but I can confirm that the same issue is present in Airflow 3.1.5 (as well as the main branch at the time of opening this PR).
Tests
Added a test covering the post-expansion state where:
The test asserts that
get_relevant_upstream_map_indexesreturns the correct upstream map index in both cases.Closes: #59289