Skip to content

Conversation

@nathadfield
Copy link
Collaborator

@nathadfield nathadfield commented Mar 1, 2024

Closes #37013

Addresses this issue referenced that prevents cluster policies from modifying attributes on dynamically mapped tasks by adding lots of additional setter methods to the MappedOperator.

@nathadfield nathadfield marked this pull request as ready for review March 1, 2024 15:42
@nathadfield nathadfield requested a review from uranusjr as a code owner March 1, 2024 15:42
@nathadfield nathadfield changed the title fix: adding lots of setter methods fix: adding MappedOperator setter methods Mar 1, 2024
@nathadfield nathadfield changed the title fix: adding MappedOperator setter methods fix: MappedOperator setter methods Mar 1, 2024
@nathadfield nathadfield changed the title fix: MappedOperator setter methods MappedOperator setter methods Mar 1, 2024
@uranusjr
Copy link
Member

uranusjr commented Mar 1, 2024

It might be easier to just make these normal instance attributes like task_id, task_group instead of living one level below.

Copy link
Contributor

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not understand the proposal of @uranusjr in the comment but from my review this looks like a good fix for the reported problem. Does not fix my other problem as discussed in #32471 but still good to go.

@jscheffl jscheffl added area:Scheduler including HA (high availability) scheduler type:bug-fix Changelog: Bug Fixes area:core labels Mar 2, 2024
@eladkal eladkal added this to the Airflow 2.8.3 milestone Mar 3, 2024
@uranusjr
Copy link
Member

uranusjr commented Mar 3, 2024

Many set-able members on MappedOperator are extracted from partial_kwargs into their own attributes on the class instead. For example, dag is popped out in OperatorPartial and set separately as its own attribute.

I can clean these up in a separate PR instead though.

@uranusjr uranusjr merged commit 1b4b73e into apache:main Mar 3, 2024
ephraimbuddy pushed a commit that referenced this pull request Mar 6, 2024
@nathadfield nathadfield deleted the fix/mapped_operator_setter_methods branch March 12, 2024 08:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core area:Scheduler including HA (high availability) scheduler type:bug-fix Changelog: Bug Fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cluster policies on mapped tasks cause broken DAGs

4 participants