Status of testing Providers that were prepared on August 11, 2023

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider alibaba: 2.5.1rc1

• Refactor: Simplify code in providers/alibaba (#33225): @eumiro

Provider amazon: 8.5.1rc1

• Get failure information on EMR job failure (#32151): @syedahsn
• Fix get_log_events() in AWS logs hook (#33290): @vincbeck
• Improve fetching logs from AWS (#33231): @vincbeck
• Refactor: Simplify code in providers/amazon (#33222): @eumiro
• Implement EventBridge enable and disable rule operators (#33226): @mahammi
• Update mypy-boto3-appflow dependency (#32930): @potiuk
• In RDS hook, used cached_property from functools not asteroid (#33133): @o-nikolas
• Use set for template_fields of EcsDeregisterTaskDefinitionOperator (#33129): @eladkal

Provider apache.beam: 5.2.1rc1

• Allow downloading requirements file from GCS in BeamRunPythonPipelineOperator (#31645): @yupbank

Provider apache.druid: 3.5.0rc1

• DruidHook add SQL-based task support (#32795): @vioao

Provider celery: 3.3.2rc1

• Add missing re2 dependency to cncf.kubernetes and celery providers (#33237): @hussein-awala
• Replace State by TaskInstanceState in Airflow executors (#32627): @hussein-awala

Provider cncf.kubernetes: 7.4.2rc1

• Add missing re2 dependency to cncf.kubernetes and celery providers (#33237): @hussein-awala
• Make the OnFinishAction enum inherit from str to support passing it to KubernetesPodOperatpor (#33228): @raphaelauv
• Refactor: Simplify code in providers/cncf (#33230): @eumiro
• Replace State by TaskInstanceState in Airflow executors (#32627): @hussein-awala

Provider common.sql: 1.7.0rc1

• Add a new parameter to SQL operators to specify conn id field (#30784): @hussein-awala

Provider databricks: 4.3.3rc1

• Add a new parameter to SQL operators to specify conn id field (#30784): @hussein-awala

Provider docker: 3.7.3rc1

• Refactor: Simplify code in providers/docker (#33232): @eumiro

Provider exasol: 4.2.3rc1

• Add a new parameter to SQL operators to specify conn id field (#30784): @hussein-awala

Provider microsoft.azure: 6.2.4rc1

• Clean microsoft azure provider by deleting the custom prefix from conn extra fields (#30558): @hussein-awala

Provider qubole: 3.4.2rc1

• Add a new parameter to SQL operators to specify conn id field (#30784): @hussein-awala

Provider snowflake: 4.4.2rc1

• Add a new parameter to SQL operators to specify conn id field (#30784): @hussein-awala

The guidelines on how to test providers can be found in

Verify providers by contributors

All users involved in the PRs:
@eladkal @o-nikolas @potiuk @hussein-awala @eumiro @syedahsn @yupbank @vioao @raphaelauv @mahammi @vincbeck

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[Lee-W]

Tested with our example DAGs with the following providers without encountering issues

• apache-airflow-providers-amazon==8.5.1rc1
• apache-airflow-providers-cncf-kubernetes==7.4.2rc1
• apache-airflow-providers-databricks==4.3.3rc1
• apache-airflow-providers-snowflake==4.4.2rc1
• apache-airflow-providers-microsoft-azure==6.2.4rc1

[potiuk]

Checked appflow #32930 - all good 👍

[o-nikolas]

Ran AWS System tests against the release candidate, all looks green: https://aws-mwaa.github.io/open-source/system-tests/version/b5a4d36383c4143f46e168b8b7a4ba2dc7c54076_8.5.1rc1.html

That covers #33133 and many of the other AWS commits (but I'll let the individual authors confirm, in case they want to do any other specific testing for their changes)

[vincbeck]

#33290 and #33231 are tested and working fine

[eladkal]

Thank you everyone.
Providers are released
I invite everyone to help improve providers for the next release, a list of open issues can be found here.