Migrate remaining provider connection UI metadata to YAML #62345
Description
Background
As part of decoupling connection form UI metadata from runtime hook imports,
we are migrating ui-field-behaviour and conn-fields from Python hook
methods (get_ui_field_behaviour() / get_connection_form_widgets()) into
declarative provider.yaml files.
This allows ProvidersManager to load connection form metadata without
importing hook classes, improving startup performance and separating UI
concerns from runtime execution.
Progress so far
- Phase 1 — Google, Amazon, Microsoft Azure, Hashicorp, Databricks, Snowflake (PR: Migrate connection UI metadata to YAML for some providers #62011)
- Phase 2 — git, ssh, sftp, docker, cncf/kubernetes, standard, grpc, opensearch, datadog, influxdb (PR: [Part 2] Migrate connection UI metadata to YAML for more providers #62109)
- Phase 3 — slack, discord, pagerduty, opsgenie, smtp, telegram, apprise (PR: [Part 3] Migrate connection UI metadata to YAML for more providers #62165)
Remaining Providers
Remaining providers
| Provider | # Connection Types | Claimed by |
|---|---|---|
airbyte |
1 | Eason09053360 |
alibaba |
4 | anishgirianish |
apache/hive |
3 | anishgirianish |
apache/iceberg |
1 | wjddn279 |
apache/kafka |
1 | wjddn279 |
apache/spark |
4 | anishgirianish |
arangodb |
1 | yjaw |
asana |
1 | yjaw |
cloudant |
1 | yjaw |
cohere |
1 | sjyangkevin |
github |
1 | yuseok89 |
jdbc |
1 | Eason09053360 |
jenkins |
1 | Eason09053360 |
mongo |
1 | haseebmalik18 |
openai |
1 | yuseok89 |
openfaas |
1 | ahilashsasidharan |
pinecone |
1 | justinpakzad |
postgres |
1 | haseebmalik18 |
qdrant |
1 | fat-catTW |
redis |
1 | yuseok89 |
salesforce |
1 | haseebmalik18 |
samba |
1 | keisuke-kawamura |
teradata |
1 | Srabasti |
trino |
1 | sjyangkevin |
weaviate |
1 | henry3260 |
yandex |
1 | Prab-27 |
ydb |
1 | henry3260 |
zendesk |
1 | henry3260 |
To claim a provider: leave a comment saying you're working on it to avoid duplicate work.
How to contribute?
Follow https://github.com/apache/airflow/blob/main/contributing-docs/23_provider_hook_migration_to_yaml.rst broadly.
- But in short you can do this in an venv with all airflow deps installed as per: https://github.com/apache/airflow/blob/main/contributing-docs/03a_contributors_quick_start_beginners.rst (cd into airflow repo and
uv sync --all-packagesshould do the trick) python scripts/tools/generate_yaml_format_for_hooks.py --provider <name>
Example:
python scripts/tools/generate_yaml_format_for_hooks.py --provider mongo
The script will print ui-field-behaviour and/or conn-fields for each connection type defined in the provider.
- Do not use the
update-yamlflag, manually make edits to the provider.yml to avoid errors - If you want to quickly verify if it works at runtime, try running this snippet:
from airflow.providers.<name>.get_provider_info import get_provider_info
for ct in get_provider_info().get('connection-types', []):
print(
ct.get('connection-type'),
'ui:', 'ui-field-behaviour' in ct,
'conn-fields:', list(ct.get('conn-fields', {}).keys())
)Once all the above instructions are done, open a PR with title: Migrate <name> connection UI metadata to YAML
Committer
- I acknowledge that I am a maintainer/committer of the Apache Airflow project.