Allow and prefer non-prefixed extra fields for AzureDataFactoryHook #27047
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boring-cyborg
bot
added
area:providers
kind:documentation
provider:microsoft-azure
potiuk
approved these changes
Oct 23, 2022
Member
|
Needs conflict resolving :( |
From airflow version 2.3, extra prefixes are not required so we enable them here.
dstandish
force-pushed
the
azure-data-factory-no-prefix
branch
from
d9a989f to
255eb78
Compare
dstandish
commented
Oct 24, 2022
pankajkoti
added a commit
to astronomer/astronomer-providers
that referenced
this pull request
Feb 27, 2023
…Azure ADF DAG failure (#899) * Support retriveing extra fields which do not have extra prefix set With the 5.0.0 release of the Microsoft Azure PR, the PR apache/airflow#27047 introduced a breaking change where while creating new connections, the ``extra__`` prefix is no longer set for extra fields in the conneciton. This issue was not identified with testing the 5.0.0 RC because, it only happens for new connections that are created. The existing connections still contain the extra fiels with the ``extra__`` prefix. Hence, the existing code which looks for the connection field with the prefix ``extra__azure_data_factory__subscriptionId`` works on the older deployment with the new provider release as the connection was created before the release, but it fails on new deployments when a fresh connection is created. To fix this, we're removing the prefix while retrieving the connection field and at the same time we're supporting previously created connections by using the same ``get_field`` method from Airflow OSS introduced in the same PR above to allow backward compatibility.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
From airflow version 2.3, extra prefixes are not required so we enable them here.