Status of testing Providers that were prepared on May 26, 2024

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider alibaba: 2.8.1rc1

• Fix oss key prefix bug in oss_task_handler, alibaba-provider (#39627): @EricGao888
  Linked issues:

Provider amazon: 8.23.0rc1

• Amazon Bedrock - Retrieve and RetrieveAndGenerate (#39500): @ferruzzi
• Introduce Amazon Comprehend Service (#39592): @gopidesupavan
• [OpenLineage] fix: empty dataset name for AthenaOperator (#39677): @kacpermuda
• Fix default value for aws batch operator retry strategy (#39608): @0x26res
• Sagemaker trigger: pass the job name as part of the event (#39671): @vincbeck
  Linked issues:
  • Linked Issue #39503: @mxuan0
• Handle task adoption for batch executor (#39590): @vincbeck
  Linked issues:
  • Linked Issue #37786: @ferruzzi
• NeptuneStopDbClusterOperator - Handle invalid cluster states (#38287): @ellisms
  Linked issues:
  • Linked Issue #38120: @ferruzzi
• Fix automatic termination issue in EmrOperator by ensuring waiter_max_attempts is set for deferrable triggers (#38658): @beobest2
• Resolving EMR deprecated warnings (#39743): @dirrao
  Linked issues:
  • Linked Issue #39485: @Taragolis

Provider apache.beam: 5.7.1rc1

• Fix deferrable mode for BeamRunJavaPipelineOperator (#39371): @MaksYermak

Provider apache.druid: 3.10.1rc1

• Pass SSL arg to all requests in DruidOperator (#39066): @danielbe11
  Linked issues:
  • Linked Issue #37673: @danielbe11

Provider apache.hive: 8.1.1rc1

• Improvising high availability field name in hive hook (#39658): @amoghrajesh

Provider apache.kafka: 1.4.1rc1

• Group id is mandatory configurtion option for confluent_kafka 2.4.0+ (#39559): @potiuk

Provider apprise: 1.3.1rc1

• Fix Apprise mypy checks added in 1.8.0 (#39580): @potiuk

Provider atlassian.jira: 2.6.1rc1

• Bump version of atlassian-python-api (#39714): @eladkal

Provider celery: 3.7.1rc1

• ECS Executor: Set tasks to RUNNING state once active (#39212): @o-nikolas
• Remove compat code for 2.7.0 - its now the min Airflow version (#39591): @jedcunningham

Provider cncf.kubernetes: 8.3.0rc1

• Add timeout when watching pod events in k8s executor (#39551): @dstandish
• Add retry logic for KubernetesCreateResourceOperator and KubernetesJobOperator (#39201): @MaksYermak
• Fix deprecated calls in cncf.kubernetes provider (#39381): @Taragolis
  Linked issues:
  • Linked Issue #38642: @Taragolis
  • Linked Issue #39356: @bmoon4
• Move Kubernetes cli to provider package (#39587): @romsharon98
• Remove compat code for 2.7.0 - its now the min Airflow version (#39591): @jedcunningham
• Replace pod_manager.read_pod_logs with client.read_namespaced_pod_log in KubernetesPodOperator._write_logs (#39112): @Lee-W

Provider common.io: 1.3.2rc1

• [openlineage] fix: OpenLineage in FileTransferOperator for Airflow 2.8 (#39755): @kacpermuda
  Linked issues:
  • Linked Issue #36410: @bolkedebruin
  • Linked Issue #39606: @potiuk

Provider common.sql: 1.14.0rc1

• Add parameters as template field for SqlSensor (#39588): @josh-fell
• DbAPiHook: Don't log a warning message if placeholder is None and make sure warning message is formatted correctly (#39690): @dabla
  Linked issues:
  • Linked Issue #39678: @romanzdk
• DbAPiHook executemany parameter of insert_rows should not be deprecated (#39630): @dabla
• Add typing for SqlSensor (#39773): @josh-fell

Provider databricks: 6.5.0rc1

• add deferrable support to DatabricksNotebookOperator (#39295): @rawwar
  Linked issues:
  • Linked Issue #39178: @pankajkoti
• [FEAT] get all failed tasks errors in exception (#39354): @gaurav7261
• Better typing for BaseOperator defer (#39742): @jedcunningham

Provider dbt.cloud: 3.8.1rc1

• Scheduler to handle incrementing of try_number (#39336): @dstandish
• Validate dbt trigger_reason field to be less than 255 characters (#38896): @boraberke
• Add (optional) dependency between dbt-cloud and openlineage providers (#39366): @tatiana

Provider docker: 3.12.0rc1

• Add args to docker service ContainerSpec (#39464): @guydriesen
• Add support to define Resources on DockerSwarmOperator (#39027): @guydriesen
• Limit requests in botocore upgrade test (#39747): @potiuk
  Linked issues:
  • Linked Issue #39740: @eladkal
• Pin requests due to incompatibility with docker-py (#39740): @eladkal

Provider elasticsearch: 5.4.1rc1

• Make elastic search index_pattern more configurable (#38423): @pankajastro
• Scheduler to handle incrementing of try_number (#39336): @dstandish

Provider fab: 1.1.1rc1

• Simplify action name retrieval in FAB auth manager (#39358): @jedcunningham
• Add jmespath as an explicit dependency (#39350): @jedcunningham

Provider google: 10.19.0rc1

• Introduce anonymous credentials in GCP base hook (#39695): @shahar1
  Linked issues:
  • Linked Issue #35755: @jvaesteves
• Remove parent_model version suffix if passed to Vertex AI operators (#39640): @e-galan
  Linked issues:
  • Linked Issue #38417: @shahar1
• Fix BigQueryCursor execute method if the location is missing (#39659): @e-galan
• Fix acknowledged functionality in deferrable mode for PubSubPullSensor (#39711): @MaksYermak
• Reroute AutoML operator links to Google Translation links (#39668): @e-galan
• Pin google-cloud-bigquery to < 3.21.0 (#39583): @pankajastro
• openlineage: remove openlineage.common dependencies in Google and Snowflake providers. (#39614): @JDarDagran
• Deprecate AutoML Tables operators (#39752): @e-galan
  Linked issues:
  • Linked Issue #38673: @molcay
• Resolve deprecation warnings in Azure FileShare-to-GCS tests (#39599): @josh-fell
  Linked issues:
  • Linked Issue #39485: @Taragolis
• [openlineage] typo: wrong OpenLineage facet key in spec (#39782): @kacpermuda
• removed stale code (#39744): @msoni1369

Provider hashicorp: 3.7.1rc1

• inherit key "verify" from env into session so that kv_client can read it properly (#38614): @chz8494

Provider microsoft.mssql: 3.7.1rc1

• use cached_property to reuse get_connections value in mssql provider (#39575): @rawwar
  Linked issues:
  • Linked Issue #39572: @rawwar
• remove extra return in get_conn of mssql provider (#39582): @rawwar

Provider openlineage: 1.8.0rc1

• Scheduler to handle incrementing of try_number (#39336): @dstandish
• [openlineage] fix: Prevent error when extractor can't be imported (#39736): @kacpermuda
  Linked issues:
  • Linked Issue #38982: @anandhimurali
• openlineage: improve spawned OpenLineage process in scheduler (#39735): @JDarDagran
  Linked issues:
  • Linked Issue #39520: @kacpermuda
• [openlineage] Update conf retrieval docstring and adjust pool_size (#39721): @kacpermuda
• openlineage: remove openlineage.common dependencies in Google and Snowflake providers. (#39614): @JDarDagran
• openlineage: use ProcessPoolExecutor over ThreadPoolExecutor. (#39235): @JDarDagran
  Linked issues:
  • Linked Issue #39232: @JDarDagran
• feat: Add custom provider runtime checks (#39609): @kacpermuda
  Linked issues:
  • Linked Issue #39336: @dstandish
  • Linked Issue #39513: @potiuk
• openlineage: notify that logged exception was caught (#39493): @mobuchowski
• [OpenLineage] Add more debug logs to facilitate debugging (#39136): @kacpermuda
• openlineage: add missing dag_state_change_process_pool_size in provider.yaml. (#39674): @JDarDagran
  Linked issues:
  • Linked Issue #39235: @JDarDagran

Provider pinecone: 2.0.0rc3

• Pinecone provider support for pinecone-client>=3 (#37307): @rawwar
  Linked issues:
  • Linked Issue #37158: @Taragolis
• Fix the pinecone system test (#39365): @sunank200
• Fix the argument type of input_vectors in pinecone upsert (#39688): @sunank200
  Linked issues:
  • Linked Issue #37307: @rawwar

Provider postgres: 5.11.1rc1

• fix: use sqlalchemy_url property in get_uri for postgresql provider (#38831): @rawwar
  Linked issues:
  • Linked Issue #38195: @Taragolis
  • Linked Issue #38187: @DEworkerDE

Provider sftp: 4.10.1rc1

• Fix SFTPSensor.newer_than not working with jinja logical ds/ts expression (#39056): @grrolland
  Linked issues:
  • Linked Issue #36629: @fpopic

Provider snowflake: 5.5.1rc1

• openlineage: remove openlineage.common dependencies in Google and Snowflake providers. (#39614): @JDarDagran
• Remove unused copy_into_postifx param from docstring (#39454): @josh-fell

Provider yandex: 3.11.1rc1

• AIP-21: yandexcloud: rename files, emit deprecation warning (#39618): @uzhastik
• yandex provider: bump version for yq http client package (#39548): @uzhastik
• add doc about Yandex Query operator (#39445): @uzhastik

All users involved in the PRs:
@pankajastro @dabla @sunank200 @e-galan @shahar1 @boraberke @JDarDagran @0x26res @ellisms @EricGao888 @grrolland @danielbe11 @tatiana @kacpermuda @gaurav7261 @gopidesupavan @mobuchowski @Lee-W @eladkal

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[uzhastik]

Provider yandex: 3.11.1rc1

Looks good, this version has been checked on my test dag and old active airflow server.
Is it possible to preview changes in doc? I have one PR in doc.

[gopidesupavan]

Hi @eladkal , i have tested #39592 , it is working fine. thanks.

[potiuk]

All my changes checked.

[shahar1]

#39695 is working fine

[EricGao888]

Hi @eladkal , I've tested #39627 and it is working fine. THX : )

[grrolland]

Provider sftp: 4.10.1rc1<https://pypi.org/project/apache-airflow-providers-sftp/4.10.1rc1> All my changes checked too. All is fine.

…

________________________________ From: Eric Gao ***@***.***> Sent: Monday, May 27, 2024 5:34:58 AM To: apache/airflow ***@***.***> Cc: Grégoire Rolland ***@***.***>; Mention ***@***.***> Subject: Re: [apache/airflow] Status of testing Providers that were prepared on May 26, 2024 (Issue #39842) Hi @eladkal<https://github.com/eladkal> , I've tested #39627<#39627> and it is working fine. THX : ) — Reply to this email directly, view it on GitHub<#39842 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AEKSCTPY7M45G5EAZMUM6GDZEKSWFAVCNFSM6AAAAABIJT4Q3OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZSGU4DAOBSGE>. You are receiving this because you were mentioned.Message ID: ***@***.***>