Status of testing Providers that were prepared on February 05, 2022

[potiuk]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 3.0.0rc1

• Rename params to cloudformation_parameter in CloudFormation operators. (#20989): @potiuk
• [SQSSensor] Add opt-in to disable auto-delete messages (#21159): @LaPetiteSouris
• Create a generic operator SqlToS3Operator and deprecate the MySqlToS3Operator. (#20807): @mariotaddeucci
• Move some base_aws logging from info to debug level (#20858): @o-nikolas
• Adds support for optional kwargs in the EKS Operators (#20819): @ferruzzi
• AwsAthenaOperator: do not generate client_request_token if not provided (#20854): @XD-DENG
• Add more SQL template fields renderers (#21237): @josh-fell
• fix: cloudwatch logs fetch logic (#20814): @ayushchauhan0811
• Alleviate import warning for EmrClusterLink in deprecated AWS module (#21195): @josh-fell
• Rename amazon EMR hook name (#20767): @vinitpayal
• Standardize AWS SQS classes names (#20732): @eladkal
• Standardize AWS Batch naming (#20369): @ferruzzi
• Standardize AWS Redshift naming (#20374): @ferruzzi
• Standardize DynamoDB naming (#20360): @ferruzzi
• Standardize AWS ECS naming (#20332): @ferruzzi
• Refactor operator links to not create ad hoc TaskInstances (#21285): @josh-fell

Provider apache.druid: 2.3.0rc1

• Add more SQL template fields renderers (#21237): @josh-fell

Provider apache.hive: 2.2.0rc1

• Add more SQL template fields renderers (#21237): @josh-fell

Provider apache.spark: 2.1.0rc1

• Add more SQL template fields renderers (#21237): @josh-fell

Provider apache.sqoop: 2.1.0rc1

• Add more SQL template fields renderers (#21237): @josh-fell

Provider cncf.kubernetes: 3.0.2rc1

• Add missed deprecations for cncf (#20031): @dimon222

Provider docker: 2.4.1rc1

• Fixes Docker xcom functionality (#21175): @ferruzzi

Provider exasol: 2.1.0rc1

• Add more SQL template fields renderers (#21237): @josh-fell

Provider google: 6.4.0rc1

• [Part 1]: Add hook for integrating with Google Calendar (#20542): @rsg17
• Add encoding parameter to GCSToLocalFilesystemOperator to fix #20901 (#20919): @danneaves-ee
• batch as templated field in DataprocCreateBatchOperator (#20905): @wsmolak
• Make timeout Optional for wait_for_operation (#20981): @MaksYermak
• Add more SQL template fields renderers (#21237): @josh-fell
• Cloudsql import links fix. (#21199): @subkanthi
• Refactor operator links to not create ad hoc TaskInstances (#21285): @josh-fell

Provider http: 2.0.3rc1

• Split out confusing path combination logic to separate method (#21247): @malthe

Provider imap: 2.2.0rc1

• Add "use_ssl" option to IMAP connection (#20441): @feluelle

Provider jdbc: 2.1.0rc1

• Add more SQL template fields renderers (#21237): @josh-fell

Provider microsoft.azure: 3.6.0rc1

• Refactor operator links to not create ad hoc TaskInstances (#21285): @josh-fell

Provider microsoft.mssql: 2.1.0rc1

• Add more SQL template fields renderers (#21237): @josh-fell

Provider microsoft.psrp: 1.1.0rc1

• PSRP improvements (#19806): @malthe

Provider mysql: 2.2.0rc1

• Add more SQL template fields renderers (#21237): @josh-fell
  (Refactor vertica_to_mysql to make it more 'mypy' friendly #20618): @potiuk

Provider oracle: 2.2.0rc1

• Add more SQL template fields renderers (#21237): @josh-fell
• Fix handling of Oracle bindvars in stored procedure call when parameters are not provided (#20720): @malthe

Provider postgres: 3.0.0rc1

• Replaces the usage of postgres:// with postgresql:// (#21205): @potiuk
• Add more SQL template fields renderers (#21237): @josh-fell
• Remove :type lines now sphinx-autoapi supports typehints (#20951): @ashb
• 19489 - Pass client_encoding for postgres connections (#19827): @subkanthi
• Amazon provider remove deprecation, second try (#19815): @uranusjr

Provider qubole: 2.1.0rc1

• Add Qubole how to documentation (#20058): @kazanzhy

Provider slack: 4.2.0rc1

• Return slack api call response in slack_hook (#21107): @pingzh
  (Fix template_fields type to have MyPy friendly Sequence type #20571): @potiuk

Provider snowflake: 2.5.0rc1

• Add more SQL template fields renderers (#21237): @josh-fell
• Fix #21096: Support boolean in extra__snowflake__insecure_mode (#21155): @mik-laj

Provider sqlite: 2.1.0rc1

• Add more SQL template fields renderers (#21237): @josh-fell

Provider ssh: 2.4.0rc1

• Add a retry with wait interval for SSH operator (#14489): @Gaurang033
• Add banner_timeout feature to SSH Hook/Operator (#21262): @potiuk

Provider tableau: 2.1.4rc1

• Squelch more deprecation warnings (#21003): @uranusjr

Provider vertica: 2.1.0rc1

• Add more SQL template fields renderers (#21237): @josh-fell

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[raphaelauv]

#21175: @ferruzzi

docker operator is buggy even with do_xcom_push=False it fail

DockerOperator(
    task_id='docker_op_tester',
    dag=dag,
    api_version='auto',
    docker_url="unix://var/run/docker.sock",
    command='/bin/echo tata',
    image='centos:latest',
    network_mode='bridge',
    mount_tmp_dir=False,
    do_xcom_push=True,
)

fail

with

[2022-02-05, 22:38:18 UTC] {taskinstance.py:1259} INFO - Executing <Task(DockerOperator): docker_op_tester> on 2022-02-03 07:00:00+00:00
[2022-02-05, 22:38:18 UTC] {standard_task_runner.py:52} INFO - Started process 325 to run task
[2022-02-05, 22:38:18 UTC] {standard_task_runner.py:76} INFO - Running: ['airflow', 'tasks', 'run', 'docker_dag', 'docker_op_tester', 'scheduled__2022-02-03T07:00:00+00:00', '--job-id', '34', '--raw', '--subdir', 'DAGS_FOLDER/docker_dag.py', '--cfg-path', '/tmp/tmpqd592sdh', '--error-file', '/tmp/tmp3e4mxwk4']
[2022-02-05, 22:38:18 UTC] {standard_task_runner.py:77} INFO - Job 34: Subtask docker_op_tester
[2022-02-05, 22:38:18 UTC] {logging_mixin.py:109} INFO - Running <TaskInstance: docker_dag.docker_op_tester scheduled__2022-02-03T07:00:00+00:00 [running]> on host 1b2f3575c860
[2022-02-05, 22:38:18 UTC] {taskinstance.py:1424} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_EMAIL=airflow@example.com
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=docker_dag
AIRFLOW_CTX_TASK_ID=docker_op_tester
AIRFLOW_CTX_EXECUTION_DATE=2022-02-03T07:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=scheduled__2022-02-03T07:00:00+00:00
[2022-02-05, 22:38:18 UTC] {docker.py:227} INFO - Starting docker container from image centos:latest
[2022-02-05, 22:38:19 UTC] {docker.py:289} INFO - tata
[2022-02-05, 22:38:19 UTC] {xcom.py:333} ERROR - Could not serialize the XCom value into JSON. If you are using pickle instead of JSON for XCom, then you need to enable pickle support for XCom in your airflow config.
[2022-02-05, 22:38:19 UTC] {taskinstance.py:1700} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1329, in _run_raw_task
    self._execute_task_with_callbacks(context)
  File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1455, in _execute_task_with_callbacks
    result = self._execute_task(context, self.task)
  File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1514, in _execute_task
    self.xcom_push(key=XCOM_RETURN_KEY, value=result)
  File "/usr/local/lib/python3.9/site-packages/airflow/utils/session.py", line 70, in wrapper
    return func(*args, session=session, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 2135, in xcom_push
    XCom.set(
  File "/usr/local/lib/python3.9/site-packages/airflow/utils/session.py", line 67, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airflow/models/xcom.py", line 100, in set
    value = XCom.serialize_value(value)
  File "/usr/local/lib/python3.9/site-packages/airflow/models/xcom.py", line 331, in serialize_value
    return json.dumps(value).encode('UTF-8')
  File "/usr/local/lib/python3.9/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/local/lib/python3.9/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.9/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.9/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type CancellableStream is not JSON serializable
[2022-02-05, 22:38:19 UTC] {taskinstance.py:1267} INFO - Marking task as FAILED. dag_id=docker_dag, task_id=docker_op_tester, execution_date=20220203T070000, start_date=20220205T223818, end_date=20220205T223819
[2022-02-05, 22:38:19 UTC] {standard_task_runner.py:89} ERROR - Failed to execute job 34 for task docker_op_tester
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/airflow/task/task_runner/standard_task_runner.py", line 85, in _start_by_fork
    args.func(args, dag=self.dag)
  File "/usr/local/lib/python3.9/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airflow/utils/cli.py", line 92, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airflow/cli/commands/task_command.py", line 298, in task_run
    _run_task_by_selected_method(args, dag, ti)
  File "/usr/local/lib/python3.9/site-packages/airflow/cli/commands/task_command.py", line 107, in _run_task_by_selected_method
    _run_raw_task(args, ti)
  File "/usr/local/lib/python3.9/site-packages/airflow/cli/commands/task_command.py", line 180, in _run_raw_task
    ti._run_raw_task(
  File "/usr/local/lib/python3.9/site-packages/airflow/utils/session.py", line 70, in wrapper
    return func(*args, session=session, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1329, in _run_raw_task
    self._execute_task_with_callbacks(context)
  File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1455, in _execute_task_with_callbacks
    result = self._execute_task(context, self.task)
  File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1514, in _execute_task
    self.xcom_push(key=XCOM_RETURN_KEY, value=result)
  File "/usr/local/lib/python3.9/site-packages/airflow/utils/session.py", line 70, in wrapper
    return func(*args, session=session, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 2135, in xcom_push
    XCom.set(
  File "/usr/local/lib/python3.9/site-packages/airflow/utils/session.py", line 67, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/airflow/models/xcom.py", line 100, in set
    value = XCom.serialize_value(value)
  File "/usr/local/lib/python3.9/site-packages/airflow/models/xcom.py", line 331, in serialize_value
    return json.dumps(value).encode('UTF-8')
  File "/usr/local/lib/python3.9/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/local/lib/python3.9/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.9/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.9/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type CancellableStream is not JSON serializable
[2022-02-05, 22:38:19 UTC] {local_task_job.py:154} INFO - Task exited with return code 1
[2022-02-05, 22:38:19 UTC] {local_task_job.py:264} INFO - 0 downstream tasks scheduled from follow-on schedule check

[kazanzhy]

Changes in Qubole provider was in documentation (#20058) and typing (#21074)
There are no code changes, therefore seems everything is OK.

[Ritika-Singhal]

#19787 requires one additional fix in here:
https://github.com/apache/airflow/blob/providers-amazon/3.0.0rc1/airflow/providers/amazon/aws/operators/glue.py
Need to modify the default value of num_of_dpus to None (which is currently set to 6).

Without this fix, it becomes inconsistent with the logic changed in the airflow/providers/amazon/aws/hooks/glue.py. It then gives the error to the user if user doesn't specify num_of_dpus=None while calling the AWSGlueJobOperator

File "/Users/ritika/opt/anaconda3/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1511, in _execute_task result = execute_callable(context=context) File "/Users/ritika/opt/anaconda3/lib/python3.7/site-packages/airflow/providers/amazon/aws/operators/glue.py", line 123, in execute create_job_kwargs=self.create_job_kwargs, File "/Users/ritika/opt/anaconda3/lib/python3.7/site-packages/airflow/providers/amazon/aws/hooks/glue.py", line 74, in __init__ raise ValueError("Cannot specify num_of_dpus with custom WorkerType") ValueError: Cannot specify num_of_dpus with custom WorkerType [2022-02-05, 16:57:41 UTC] {local_task_job.py:154} INFO - Task exited with return code 1

[mariotaddeucci]

#20807 all good.
#20506 was tested on #20615 (amazon 2.6.0)

[Ritika-Singhal]

For an additional fix for #19787, I have created another pull request #21353
All other functionalities in #19787 works as expected.

[potiuk]

Ok. Still worth testing Amazon changes on this version but I will remove Amazon + Docker (if confirmed) and prepare RC2 right after we release all others and get fixes merged. Thanks @Ritika-Singhal @raphaelauv !

[potiuk]

I also removed some of the "no-need-to-test" changes (and 2.6.0 amazon changes - which was a bug in generation of the issue - sorry for that.

[rsg17]

Hi @potiuk - It is my first time doing this testing: Are there any specific steps we follow?

My PR adds a hook for Google Calendar.

[potiuk]

Hi @potiuk - It is my first time doing this testing: Are there any specific steps we follow?

Good question. Depends on your "local" environment. But what I would do, is to use:

1. Start airflow 2.2.3 in breeze

./breeze start-airflow --use-airflow-version 2.2.3 --backend postgres --db-reset  --load-default-connections 

This will start our dockerized development environment with Airflow 2.2.3 installed and open 4 terminals: triggerer, scheduler, webserver and "bash console".

2. update the package to RC:

In the console:

pip install apache-airflow-providers-google==6.4.0rc1 

Then restarting scheduler/webserver (by Ctrl+C) followed by "up cursor" to go back and run previous command

3. Prepare a test dag using calendar api and run it

For that you likely need to configure the "google_default_connection" to include your credentials. You can put your dags in "files/dags" folder of your Airlfow sources (the "files" folder is mounted to inside the docker) and they should be scanned/visible in webserver

[malthe]

@potiuk – Oracle and PSRP providers all good.

[josh-fell]

@potiuk Part of #21237 was adding 3 new lexers for the SQL rendering in the UI. For the providers that use the new lexers, the rendering won't be as readable as it could be because the configured lexer doesn't exist yet (i.e. the rendering becomes a Python type rendering). Do we want to wait on releasing #21237 at all until those lexers are available or not push the SQL template_fields_renderers change for those particular providers yet?

For example, there is a new postgresql lexer and added to the RedshiftSqlOperator (among others). Without the new lexer in core Airflow, the rendering of the SQL statement looks like this:

Instead of:

[potiuk]

Good point Josh. I think this one might be a bit tricky, I do not think it is "blocker" - the rendering is not nice but it does not "break" anything.

However I thought it might be a nice way to incentivize people to migrate to 2.2.4 if we also cherry-pick the lexer to 2.2.4. Then we could telll them "migrate to 2.2.4 to get it nicer". This is a new feature - of course - so technically we shoud add it in 2.3.0. So I am a bit torn here.

There are quite a number of those providers that are only released because of this renderer so we can easily skip them from this release and release them when we release 2.3.0

@jedcunningham @josh-fell @wdyt?

[potiuk]

The second option is to indeed drop all the "21237" providers an release them in RC2 with some "conditional" code that will check if the lexer is there. After thinking a bit I think that would be much "cleaner" solution and I am leaning towards this option.

smth like:

template_fields_renderer = {//}.update({'sql': 'sql'} if 'sql' in get_attr_renderer() else {})