Make conn id parameters templated in GenericTransfer and also allow passing hook parameters like in BaseSQLOperator #42891
Conversation
…conn_id's for GenericTransfer and also allow passing hook params for each connection in GenericTransfer
…GenericTransfer and BaseSQLOperator
boring-cyborg
bot
added
area:core-operators
…r as it's part of Airflow itself
…n GenericTransfer as it's part of Airflow itself
|
Looks like there is some problem with minimum version of one of microsoft dependencies. Bringing it down, brings opentelemetry dependency down and this seems to break imports. Unfold the failng test output for microsoft.azure and you will see it: https://github.com/apache/airflow/actions/runs/11299097052/job/31429704576?pr=42891#step:7:6196 |
Seems to be fine now after I updated the branch |
|
There are some compatibility Airflow 3 / Airflow 2 issues that need to be addressed :) |
…viders-common-sql version 1.18.0 or higher
All issues fixed |
Today I've discovered that the conn id parameters (source_conn_id and destination_conn_id) aren't templated in the GenericTransfer. This PR will also allow passing hook parameters like in BaseSQLOperator. Refactored the get_hook method in BaseOperator so that in the future it can be used in a generic way, added tests to remove backward compatibility code for Airflow 2.8.x once min version of provider is Aiflow 3.0.0 or higher.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.