Skip to content

Conversation

@SameerMesiah97
Copy link
Contributor

Description

Fix upstream map index resolution for dynamically mapped task groups when placeholder task instances are replaced during expansion.

After expansion, the upstream placeholder (map_index = -1) is replaced by the first expanded task instance (map_index = 0). Downstream task instances that are still unexpanded may continue to reference the placeholder, causing incorrect upstream dependency resolution.

This change introduces a small helper (resolve_placeholder_map_index) that is invoked during upstream map index resolution to correctly handle this transition, while preserving existing behavior for already-expanded downstream tasks.

Rationale

Placeholder task instances are a temporary pre-expansion representation. Once expansion occurs, they are no longer valid references, but downstream resolution logic may still encounter them. Handling this transition explicitly prevents downstream tasks from treating completed upstream work as unresolved.

This could cause implicit dependencies to be skipped during DAG evaluation under certain trigger rules (such as NONE_FAILED_MIN_ONE_SUCCESS), particularly when multiple parallel task streams within the MappedTaskGroup converge on a single downstream task outside the group. As a result, DAG runs could be incorrectly marked as failed or skipped despite valid upstream execution.

Please refer to issue #59289 for more context. This PR was opened in response to that. The author of the issue reported the bug in Airflow 3.0.6 but I can confirm that the same issue is present in Airflow 3.1.5 (as well as the main branch at the time of opening this PR).

Tests

Added a test covering the post-expansion state where:

  • The upstream placeholder has been expanded.
  • The downstream task instance is either unexpanded or expanded.

The test asserts that get_relevant_upstream_map_indexes returns the correct upstream map index in both cases.

Closes: #59289

@SameerMesiah97
Copy link
Contributor Author

@ashb @XD-DENG

This is just a soft reminder for review. It fixes a genuine bug (please refer to issue #59289) that subtly breaks DAGs under certain mapped task group setups. I believe it should be addressed before the next release.

@potiuk
Copy link
Member

potiuk commented Dec 28, 2025

Also @uranusjr can help with review - there ias related WIP he eworks on now I think

@SameerMesiah97 SameerMesiah97 force-pushed the 59289-DTM-Placeholder-Resolution branch 2 times, most recently from 3b16811 to cee1968 Compare January 6, 2026 19:08
@SameerMesiah97
Copy link
Contributor Author

@uranusjr

Since, according to @potiuk your recent work is adjacent to what is being addressed in this PR, I am tagging you as a soft reminder for review and/or closing #59289 if you feel that your work has addressed it.

@SameerMesiah97 SameerMesiah97 force-pushed the 59289-DTM-Placeholder-Resolution branch from cee1968 to 630de66 Compare January 18, 2026 11:06
@SameerMesiah97
Copy link
Contributor Author

I have checked the main branch using a variety of DAG shapes following the same theme as the one in the bug report and the issue is still present. So it appears any recent work in this specific area (that is separate from this PR) which has been merged into main, has not resolved the bug.

As of this comment, I cannot see any open PRs that could resolve the issue (anyone reading this comment is free to correct me).

This PR has been rebased and is ready for review.

@eladkal eladkal added this to the Airflow 3.1.7 milestone Jan 19, 2026
@eladkal eladkal added area:core area:dynamic-task-mapping AIP-42 backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch labels Jan 19, 2026
@eladkal eladkal requested a review from uranusjr January 19, 2026 07:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core area:dynamic-task-mapping AIP-42 backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dynamic Task Mapping Skips Before Upstream Has Started

4 participants