Status of testing Providers that were prepared on February 08, 2023

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 7.2.0rc1

• Add option to wait for completion on the EmrCreateJobFlowOperator (#28827): @BasPH
• Add transfer operator S3 to (generic) SQL (#29085): @maggesssss
• add retries to stop_pipeline on conflict (#29077): @vandonr-amz
• Add log for AWS Glue Job Console URL (#28925): @IAL32
• Enable trigger logging in webserver (#27758): @dstandish
• fix: type annotation for num_of_dpus in GlueJobHook (#29176): @wonhyeongseo
• Fix typo in DataSyncHook boto3 methods for create location in NFS and EFS (#28948): @Taragolis
• Decrypt SecureString value obtained by SsmHook (#29142): @Taragolis
• log the observed status in redshift sensor (#29274): @vandonr-amz
• Use thin/passthrough hook instead of one-liner hook method (#29252): @o-nikolas
• Move imports in AWS SqlToS3Operator transfer to callable function (#29045): @joarobles
• introduce base class for EKS sensors (#29053): @vandonr-amz
• introduce a method to convert dictionaries to boto-style key-value lists (#28816): @vandonr-amz
• Update provide_bucket_name() decorator to handle new conn_type (#28706): @RachitSharma2001
• uniformize getting hook through cached property in aws sensors (#29001): @vandonr-amz
• Use boto3 intersphinx inventory in documentation/docstrings. (#28945): @Taragolis

Provider apache.beam: 4.2.0rc1

• Add support for running a Beam Go pipeline with an executable binary (#28764): @johannaojeling
• Deprecate delegate_to param in GCP operators and update docs (#29088): @shahar1

Provider apache.hive: 5.1.2rc1

• Fixed MyPy errors introduced by new mysql-connector-python (#28995): @potiuk

Provider arangodb: 2.1.1rc1

• Fix ArangoDB static checks (#28820): Not require testing

Provider cncf.kubernetes: 5.2.0rc1

• Add deferrable mode to KubernetesPodOperator (#29017): @VladaZakharova
• Allow setting the name for the base container within K8s Pod Operator (#28808): @csm10495
• Patch only single label when marking KPO checked (#29279): @jedcunningham

Provider dbt.cloud: 3.0.0rc1

• Drop Connection.schema use in DbtCloudHook (#29166): @josh-fell
• Allow downloading of dbt Cloud artifacts to non-existent paths (#29048): @josh-fell
• Add deferrable mode to DbtCloudRunJobOperator (#29014): @phanikumv
• Provide more context for trigger_reason in DbtCloudRunJobOperator (#28994): @josh-fell

Provider elasticsearch: 4.4.0rc1

• Enable trigger logging in webserver (#27758): @dstandish

Provider ftp: 3.3.1rc1

• FTP operator has logic in init (#29073): @snjypl

Provider google: 8.9.0rc1

• Add deferrable capability to existing DataprocDeleteClusterOperator (#29349): @pankajkoti
• Add deferrable mode to dataflow operators (#27776): @MrGeorgeOwl
• Add deferrable mode to DataprocCreateBatchOperator (#28457): @bkossakowska
• Add deferrable mode to DataprocCreateClusterOperator and (#28529): @bkossakowska
• Add deferrable mode to MLEngineStartTrainingJobOperator (#27405): @VladaZakharova
• Add deferrable mode to DataFusionStartPipelineOperator (#28690): @VladaZakharova
• Add deferrable mode for Big Query Transfer operator (#27833): @MrGeorgeOwl
• Add support for write_on_empty in BaseSQLToGCSOperator (#28959): @vchiapaikeo
• Add DataprocCancelOperationOperator (#28456): @bkossakowska
• Enable trigger logging in webserver (#27758): @dstandish
• Auto ML assets (#25466): @MaksYermak
• Fix GoogleDriveHook writing files to trashed folders on upload v2 (#29119): @vchiapaikeo
• fix Google provider CHANGELOG.rst (#29122): Not require testing
• fix Google provider CHANGELOG.rst (#29114): Not require testing
• Keyfile dict can be dict not str (#29135): @dstandish
• GCSTaskHandler may use remote log conn id (#29117): @dstandish
• Deprecate delegate_to param in GCP operators and update docs (#29088): @shahar1

Provider microsoft.azure: 5.2.0rc1

• Enable trigger logging in webserver (#27758): @dstandish
• Fix params rendering in AzureSynapseHook Python API docs (#29041): @josh-fell
• Deprecate delegate_to param in GCP operators and update docs (#29088): @shahar1

Provider mysql: 4.0.1rc1

• Fixed MyPy errors introduced by new mysql-connector-python (#28995): @potiuk

Provider presto: 4.2.2rc1

• Deprecate delegate_to param in GCP operators and update docs (#29088): @shahar1

Provider sftp: 4.2.2rc1

• Fix SFTP operator's template fields processing (#29068): @potiuk
• FTP operator has logic in init (#29073): @snjypl

Provider snowflake: 4.0.3rc1

• Provide the missing connection id for the parent classes of snowflake operators (#29211): @hussein-awala
• Snowflake Provider Package - hide host from UI (#29208): @dwreeves

Provider tableau: 4.1.0rc1

• (#29353) feat: TableauOperator.template_fields = find, match_with (#29360): @seunggabi

Provider trino: 4.3.2rc1

• Deprecate delegate_to param in GCP operators and update docs (#29088): @shahar1

The guidelines on how to test providers can be found in

Verify providers by contributors

Contributors are also encouraged to cast vote in the mailing list thread.

All users involved in the PRs:
@BasPH @VladaZakharova @bkossakowska @pankajkoti @phanikumv @MaksYermak @RachitSharma2001 @vandonr-amz @IAL32 @raphaelauv @Taragolis @dwreeves @potiuk @snjypl @dstandish @vchiapaikeo @eladkal @shahar1
@jedcunningham @csm10495 @josh-fell @joarobles @johannaojeling @o-nikolas @wonhyeongseo @seunggabi @MrGeorgeOwl @hussein-awala @maggesssss

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[pankajkoti]

Thanks @eladkal . I tested my PR #29349 which is part of Provider google: 8.9.0rc1 and it works fine.

[seunggabi]

Thanks @eladkal .
I tested a minute ago.

#29360: @seunggabi
https://pypi.org/project/apache-airflow-providers-tableau/4.1.0rc1/

It was works.

from airflow.providers.tableau.operators.tableau import TableauOperator

...

task_refresh_workbook_blocking = TableauOperator(
    resource="workbooks",
    method="refresh",
    match_with="{{ params.match_with }}",
    find="{{ params.find }}",
    blocking_refresh=True,
    task_id="refresh_tableau_workbook_blocking",
    dag=dag,
)

[johannaojeling]

#28764 is tested with the Beam RC and works as expected

[csm10495]

#28808 still seems legit

[vchiapaikeo]

#28959 looks good

write_on_empty=False (default)

write_on_empty=True

[o-nikolas]

I added code to kill the process while running to test the emr on_kill changes, and the cleanup code ran successfully (and I confirmed the resource was deleted in my AWS account):

INFO  [airflow.task.operators] Job flow with id j-1ADYKTIV4THO5 created
INFO  [airflow.task.operators] SLEEPING NOW, KILL THE PROCESS TO TEST ON_KILL
INFO  [airflow.task.operators] TRYING TO KILL MYSELF
ERROR [airflow.task] Received SIGTERM. Terminating subprocesses.
INFO  [airflow.task.operators] Terminating job flow j-1ADYKTIV4THO5

While I was there, I tested the wait_for_completion changes on the same operator. It worked successfully:

INFO  [airflow.task] Executing <Task(EmrCreateJobFlowOperator): create_job_flow> on 2021-01-01 00:00:00+00:00
INFO  [airflow.task] Exporting env vars: AIRFLOW_CTX_DAG_OWNER='airflow' AIRFLOW_CTX_DAG_ID='example_emr' AIRFLOW_CTX_TASK_ID='create_job_flow' AIRFLOW_CTX_EXECUTION_DATE='2021-01-01T00:00:00+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='backfill__2021-01-01T00:00:00+00:00'
INFO  [airflow.task.operators] Creating job flow using aws_conn_id: aws_default, emr_conn_id: emr_default
INFO  [airflow.hooks.base] Using connection ID 'emr_default' for task execution.
INFO  [airflow.hooks.base] Using connection ID 'aws_default' for task execution.
INFO  [botocore.credentials] Found credentials in environment variables.
INFO  [airflow.task.operators] Job flow with id j-2VDGDHAR3ASCT created
INFO  [airflow.providers.amazon.aws.utils.waiter] Waiting for job flow to be finished.
INFO  [airflow.providers.amazon.aws.utils.waiter] Waiting for job flow to be finished.
INFO  [airflow.providers.amazon.aws.utils.waiter] Waiting for job flow to be finished.
INFO  [airflow.providers.amazon.aws.utils.waiter] Waiting for job flow to be finished.
INFO  [airflow.providers.amazon.aws.utils.waiter] Waiting for job flow to be finished.

I marked both changes as tested above in the task list.

[josh-fell]

Thanks for organizing @eladkal!

All of the dbt Cloud changes look good as well as the Python API docs change in #29041.

[eladkal]

Thank you everyone.
Providers are released
I invite everyone to help improve providers for the next release, a list of open issues can be found here.