Get Operator Extra Links working again

[amoghrajesh]

Operator links do not work in Airflow 3.

• The plan is to change the interface in a way that webserver doesn't have to execute the user code: get_link
• We will push an xcom during task runtime to the XCOM backend. The key will be as defined in the operator link
• While reading from the webserver, instead of reading from the get_link method, XCom is read.

[eladkal]

We will push an xcom during task runtime to the XCOM backend. The key will be as defined in the operator link

How is this going to work when user created several extra links for the same operator?

[amoghrajesh]

We will be adding "xcom_key" at the interface level for operator links. For example:

class DummyTestOperator(BaseOperator):
    # We need to ignore type check here due to 2.7 compatibility import
    operator_extra_links = (  # type: ignore[assignment]
        RaiseErrorLink(),
        NoResponseLink(),
        FooBarLink(),
        AirflowLink(),
    )

For such cases, each of the links here are attached to this operator and they have XCOM keys of their own.
So while doing a lookup from webserver, we will fetch all the links pertaining to the operator.

We also are storing at the serdag level to assist with this

[hanxdatadog]

Seeing an issue on this topic. The serialized XComOperatorLink is not respecting custom xcom backend but it looks like it is by design using raw XComModel (DB backend)

[potiuk]

Seeing any issue on this topic that the serialized XComOperatorLink is not respecting custom xcom backend.

I read it three times @hanxdatadog and I do not understand what you mean. I don't even know if you are asking to change something, endorse the change, or complain about it.

I suggest - if you have an actionable proposal, open an issue describing what your issue is, possibly pointing to the PR. This PR is now "merged" - so if it causes any issue, open an issue, with reproducible case and link to that PR.

[hanxdatadog]

Seeing any issue on this topic that the serialized XComOperatorLink is not respecting custom xcom backend.

I read it three times @hanxdatadog and I do not understand what you mean. I don't even know if you are asking to change something, endorse the change, or complain about it.

I suggest - if you have an actionable proposal, open an issue describing what your issue is, possibly pointing to the PR. This PR is now "merged" - so if it causes any issue, open an issue, with reproducible case and link to that PR.

Apologies for the confusion. @potiuk. I had a typo, just edited it and add more explanations. I opened a discussion earlier. #59914. Basically we are using object storage xcom backend but the operator link expects only DB backend xcom pushes.
I was trying something out changes locally. But after reading more code, now I see custom xcom backend is only used at the execution layer. We can follow up more on that discussion.