Skip to content

Fix assets used only as inlets being incorrectly orphaned (#58303) #58368

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
uranusjr merged 5 commits into from
Dec 3, 2025
Merged

Fix assets used only as inlets being incorrectly orphaned (#58303) #58368

uranusjr merged 5 commits into from
Dec 3, 2025

Conversation

@mykola-shyshov
Copy link
Contributor

@mykola-shyshov mykola-shyshov commented Nov 16, 2025

Fix assets with only inlet references being incorrectly orphaned (#58303)

Assets that were referenced only in task inlets were incorrectly marked as orphaned and removed from the AssetActive table. This caused a race condition where validate_inlets_and_outlets would intermittently fail with AirflowInactiveAssetInInletOrOutletException.

The orphanage detection logic only checked DagScheduleAssetReference and TaskOutletAssetReference, but did not include TaskInletAssetReference. This meant assets used exclusively as task inputs were considered unreferenced and removed from active tracking.

WARN: may be it is intentionally!

Changes:

  • Include TaskInletAssetReference in orphanage detection query
  • Add regression tests for inlet-only asset activation

Fixes #58303
Screenshot 2025-11-16 at 17 48 38

@boring-cyborg boring-cyborg bot added the area:Scheduler including HA (high availability) scheduler label Nov 16, 2025
@mykola-shyshov mykola-shyshov force-pushed the fix-58303/flaky-AirflowInactiveAssetInInletOrOutlet branch 2 times, most recently from 327fa7d to 76d0cd8 Compare November 16, 2025 19:15
@KoviAnusha
Copy link
Contributor

KoviAnusha commented Nov 16, 2025 •
edited
Loading

@mykola-shyshov, great work on this fix. One question while reviewing: do we expect any noticeable perf impact when counting references for large asset sets? I assume minimal, but curious if we’ve thought about it.

Also, confirming my understanding, this ensures inlet-only assets remain active and don’t get incorrectly dropped from AssetActive, correct?

Maybe in the future we could add a test with mixed inlet/outlet references in the same DAG to fully cover real-world patterns, not necessary for this PR though.

@kacpermuda
Copy link
Contributor

kacpermuda commented Nov 17, 2025

We should probably also backport this fix to AF 3.1 ?

@mykola-shyshov
Fix assets used only as inlets being incorrectly orphaned (apache#58303)
64ee24a 
Assets referenced only in task inlets were removed from AssetActive, causing intermittent validation failures.

Fixes apache#58303
@mykola-shyshov mykola-shyshov force-pushed the fix-58303/flaky-AirflowInactiveAssetInInletOrOutlet branch from e06e9f5 to 64ee24a Compare November 17, 2025 16:29
@mykola-shyshov
Copy link
Contributor Author

mykola-shyshov commented Nov 17, 2025 •
edited
Loading

@KoviAnusha

great work on this fix. One question while reviewing: do we expect any noticeable perf impact when counting references for large asset sets? I assume minimal, but curious if we’ve thought about it.

thanks. Unfortunately, I can not predict how much it may impact. Foreign index is there so it should be fine. Delegating this question to reviewers.

Also, confirming my understanding, this ensures inlet-only assets remain active and don’t get incorrectly dropped from AssetActive, correct?

Yes

@kacpermuda

We should probably also backport this fix to AF 3.1 ?

If something need to be done from my side direct me.

@kacpermuda
Copy link
Contributor

kacpermuda commented Nov 17, 2025

I think the maintainer has to set a proper label on the PR so it's backported, so let's wait for someone to review the PR.

kacpermuda
kacpermuda approved these changes Nov 19, 2025
View reviewed changes
Copy link
Contributor

@kacpermuda kacpermuda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, @ashb or @uranusjr maybe you could have a look and merge this when you have a moment ? Seems like a small bug with an easy fix

uranusjr
uranusjr reviewed Nov 20, 2025
View reviewed changes
uranusjr
uranusjr reviewed Nov 20, 2025
View reviewed changes
@kacpermuda kacpermuda mentioned this pull request Nov 25, 2025
Merged
@kacpermuda
Copy link
Contributor

kacpermuda commented Nov 26, 2025

@uranusjr Does the fix look good now after changes? Can we merge it?

@ephraimbuddy ephraimbuddy added the backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch label Dec 1, 2025
Lee-W
Lee-W approved these changes Dec 1, 2025
View reviewed changes
Copy link
Member

@Lee-W Lee-W left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few nits but non-blocking

@uranusjr uranusjr merged commit 24e682c into apache:main Dec 3, 2025
118 checks passed
github-actions bot pushed a commit that referenced this pull request Dec 3, 2025
@mykola-shyshov
[v3-1-test] Fix assets used only as inlets being incorrectly orphaned (…
103dad1 
…#58303) (#58368)

(cherry picked from commit 24e682c)

Co-authored-by: Mykola Shyshov <mykola.shyshov@gmail.com>
@github-actions
Copy link

github-actions bot commented Dec 3, 2025

Backport successfully created: v3-1-test

Status Branch Result
✅ v3-1-test PR Link

github-actions bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Dec 3, 2025
@mykola-shyshov
[v3-1-test] Fix assets used only as inlets being incorrectly orphaned (…
a396a1c 
…apache#58303) (apache#58368)

(cherry picked from commit 24e682c)

Co-authored-by: Mykola Shyshov <mykola.shyshov@gmail.com>
Lee-W pushed a commit that referenced this pull request Dec 3, 2025
@github-actions @mykola-shyshov
[v3-1-test] Fix assets used only as inlets being incorrectly orphaned (…
5e0d53a 
…#58303) (#58368) (#58986)

Co-authored-by: Mykola Shyshov <mykola.shyshov@gmail.com>
ephraimbuddy pushed a commit that referenced this pull request Dec 3, 2025
@github-actions @mykola-shyshov @ephraimbuddy
[v3-1-test] Fix assets used only as inlets being incorrectly orphaned (…
efcf97c 
…#58303) (#58368) (#58986)

Co-authored-by: Mykola Shyshov <mykola.shyshov@gmail.com>
itayweb pushed a commit to itayweb/airflow that referenced this pull request Dec 6, 2025
@mykola-shyshov @itayweb
Fix assets used only as inlets being incorrectly orphaned (apache#58303…
93724ef 
…) (apache#58368)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Lee-W Lee-W Lee-W approved these changes

@uranusjr uranusjr uranusjr approved these changes

@ashb ashb Awaiting requested review from ashb ashb is a code owner

@XD-DENG XD-DENG Awaiting requested review from XD-DENG XD-DENG is a code owner

+1 more reviewer

@kacpermuda kacpermuda kacpermuda approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area:Scheduler including HA (high availability) scheduler backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

AirflowInactiveAssetInInletOrOutletException is flaky

6 participants

@mykola-shyshov @KoviAnusha @kacpermuda @uranusjr @Lee-W @ephraimbuddy