Implements AwsBaseOperator and AwsBaseSensor

[Taragolis]

Continuation #24246

This base operators and sensors might help to resolve inconsistency in multiple operators and sensors:

1. Some of current sensors/operators accept only aws_conn_id
2. Some of current sensors/operators accept region instead of region_name

In additional this implementation propagate additional parameters into the hook:

1. Operator/Sensor botocore_config -> Hook config
2. Operator/Sensor verify -> Hook verify

Note

AwsBaseSensor created by "old methodic"

1. Copy code from AwsBaseOperator
2. Paste in new module

I've tried to create Mixin class however it not works well breaks something in BaseOperator
If someone know how to implements it by DRY, without use additional Metaclasses and/or class decorators feel free to suggest.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[Taragolis]

It is also might be a good idea to make this classes internal only.

[Taragolis]

And wondering about module naming because I a bit confused:

Suggestion in old PR use base_aws: #24246 (comment)
Naming for Base Trigger: airflow.providers.amazon.aws.triggers.base.AwsBaseWaiterTrigger

cc @ferruzzi @vincbeck @o-nikolas

There are only two hard things in Computer Science: cache invalidation and naming things

[vincbeck]

Agree, naming is hard xD. I am fine with base_aws.py. It could be also:

• base_operator.py
• base.py

[o-nikolas]

It is also might be a good idea to make this classes internal only.

+1

[Taragolis]

This changes also required to make changes in airflow.providers.amazon.aws.triggers.base.AwsBaseWaiterTrigger as follow-up.

That is also why I make accept botocore_config as dictionary and not botocore.config.Config object

[ferruzzi]

Left some suggestions. We've discussed an AwsBaseOperator a few times and kept putting it off, there may be some more suggestions coming once I check my notes on what I was going to put into it.

As always, thanks for your thought and effort on the provider package contribution. 👍

[ferruzzi]

I think this is repeating itself, but maybe I'm missing something.

[o-nikolas]

Using the default AWS Connection id aws_default (as long as it is created and remains untouched by the user) also uses the default boto3 behaviour. Might be worth mentioning.

[Taragolis]

Maybe we could try to rewrite it to something shorter with reference to AWS Connection, like we have in other providers, e.g. Slack

airflow/airflow/providers/slack/operators/slack_webhook.py

Lines 42 to 43 in ed6a4fd

	:param slack_webhook_conn_id: :ref:`Slack Incoming Webhook <howto/connection:slack>`
	connection id that has Incoming Webhook token in the password field.

Example in Auto API:
https://airflow.apache.org/docs/apache-airflow-providers-slack/8.1.0/_api/airflow/providers/slack/operators/slack_webhook/index.html

[ferruzzi]

Looks like my comments have all been addressed. Thanks.

[eladkal]

I assume followup PR would be to change AWS operators/hooks to inherit from the base classes?

We need also doc update to explain how to set the botocore parameters

[eladkal]

You need to move these to the BASE_CLASSES

[Taragolis]

I was a bit blind 🤣

[Taragolis]

I assume followup PR would be to change AWS operators/hooks to inherit from the base classes?

Yep

We need also doc update to explain how to set the botocore parameters

I think it would be a good idea to add generic information here: https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/operators/index.html as soon as we change existed Operators, so them will have same sets of generic parameters:

• aws_conn_id: Operators which intends to use AwsBaseHook already have
• region_name: Some of operators use this parameter
• verify: Only one use right now
• botocore_config: None of of the operators use it right now

[vincbeck]

Agree, naming is hard xD. I am fine with base_aws.py. It could be also:

• base_operator.py
• base.py

[vincbeck]

Out of curiosity. Why from None?

[Taragolis]

Just for skip previous traceback, e.g. if developer set aws_hook_class then it failed with TypeError on issubclass and traceback will contain information about error which raised from issubclass

[vincbeck]

I see, thanks for the explanation

[ferruzzi]

I had to research that one too, I have never seen that in use. It's kinda handy.

[Taragolis]

In 126a1f5 split logic into two classes, for reduce repeatable code in AwsBaseOperator and AwsBaseSensor and in additional this might help to use in other places, e.g. Notifiers. Potentially it also could be use in AwsBaseWaiterTrigger however it use hook as a method, rather than property

[o-nikolas]

Regarding this, should we change the class name to _AwsBaseOperator?

Same for the Sensor and mixin classes

[Taragolis]

There is kind of tradeoff between common Python's agreement about internal variables/classes/methods and airflow public models. _ shows that is internal stuff however IDE doesn't like this imports even with the same package (I guess static checks is fine) in the other hand we have some methods and class in core which do no prefixed by _ but have a description and :meta private:.

There is also maybe a good idea to make this stuff public after a while for create custom operators, but for now I would rather keep it internal only

In the end no one could stop end users to use this stuff directly

[o-nikolas]

Using the default AWS Connection id aws_default (as long as it is created and remains untouched by the user) also uses the default boto3 behaviour. Might be worth mentioning.

[syedahsn]

Looks good to me. I'd be interested to see how much this affects new and existing operators and sensors.

[eladkal]

LGTM!