Skip to content

Conversation

@gopidesupavan
Copy link
Member

@gopidesupavan gopidesupavan commented Apr 18, 2025

In AF2 , get_operator_extra_links was used to get the extra links, and they were handled in webserver. In AF3 a new XComOperatorLink class added and extra links are no longer processed in the webserver. Instead when a taks runs, the get_link result is pushed to XCom . The webserver then reads it from the XCom.

For plugins defined links, They are converted to XComOperatorLink during deserialization so the webserver can read them too. https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/serialization/serialized_objects.py#L1607


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@gopidesupavan
Copy link
Member Author

Added these links to standard provider section #49447

@gopidesupavan
Copy link
Member Author

In AF2 this is being used part of _deserialize_operator_extra_links . Is there any reason not using this method in AF3? to my understand for AF3 its referring new class XComOperatorLink could this be reason ?

@amoghrajesh
Copy link
Contributor

Yeah @gopidesupavan thats the reason. Now the operator extra links do not run in the webserver anymore.

We push the value generated by get_link function to xcom when a task is running instead (for operator_extra_links at operator / task level) and the webserver reads that off. For operator links in plugins defined, during deserialisation, we convert that object to a xcomoperatorlink instead and webserver just reads the xcom

@gopidesupavan
Copy link
Member Author

Yeah @gopidesupavan thats the reason. Now the operator extra links do not run in the webserver anymore.

We push the value generated by get_link function to xcom when a task is running instead (for operator_extra_links at operator / task level) and the webserver reads that off. For operator links in plugins defined, during deserialisation, we convert that object to a xcomoperatorlink instead and webserver just reads the xcom

Thanks yeah, Alright we can cleanup these methods.

Copy link
Contributor

@amoghrajesh amoghrajesh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah i think we can do this now. Looks fine to me, but would like a second set of eyes too in case we miss something obvious.

@gopidesupavan
Copy link
Member Author

Yeah i think we can do this now. Looks fine to me, but would like a second set of eyes too in case we miss something obvious.

fine :)

@gopidesupavan gopidesupavan force-pushed the cleanup-operator-extra-links branch from fe0532b to fcc3382 Compare April 22, 2025 17:17
@gopidesupavan gopidesupavan force-pushed the cleanup-operator-extra-links branch from fcc3382 to 47e8084 Compare April 24, 2025 21:15
@gopidesupavan
Copy link
Member Author

Anyone would like to have a look into this?

@kaxil
Copy link
Member

kaxil commented Apr 25, 2025

Could you add a tldr in PR description of What's the problem or if they are unused -- i.e. motivation behind the change

Copy link
Member

@kaxil kaxil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once you add PR description, you are good to go :)

@kaxil kaxil changed the title Cleanup _OPERATOR_EXTRA_LINKS in serialized_objects Remove unused _OPERATOR_EXTRA_LINKS in serialized_objects Apr 25, 2025
@gopidesupavan
Copy link
Member Author

Could you add a tldr in PR description of What's the problem or if they are unused -- i.e. motivation behind the change

Yeah added pretty much the understand from the amogh comment i have added there :)

@gopidesupavan gopidesupavan merged commit b5efb47 into apache:main Apr 25, 2025
51 checks passed
jroachgolf84 pushed a commit to jroachgolf84/airflow that referenced this pull request Apr 30, 2025
@gopidesupavan gopidesupavan deleted the cleanup-operator-extra-links branch May 28, 2025 20:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants