Skip to content

Conversation

@vandonr-amz
Copy link
Contributor

also caught some hooks that were created in the ctor.

@boring-cyborg boring-cyborg bot added area:providers provider:amazon AWS/Amazon - related issues labels Jan 18, 2023
Copy link
Contributor

@ferruzzi ferruzzi Jan 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of these files you are passing region to the hook and others are not. Should we standardize that as well?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mostly kept the existing behavior. I don't know what is a real use case for passing the region, maybe that depends on the operator/sensor ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Somewhere in the past I have an idea to keep Sensors/Operators more consistent and provide all of supported parameters in Hook.
I think it safe to pass region_name because we set it to None by default in Operators/Sensors.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW , it is not related to this PR, but we have a 3 different hooks which is wraps around Glue.Client 🤦

  • airflow.providers.amazon.aws.hooks.glue.GlueJobHook
  • airflow.providers.amazon.aws.hooks.glue_catalog.GlueCatalogHook
  • airflow.providers.amazon.aws.hooks.glue_crawler.GlueCrawlerHook

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW , it is not related to this PR, but we have a 3 different hooks which is wraps around Glue.Client facepalm

Yeah, there is also a BatchWaiterHook which is an oddball too. Some Day ™️

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In one recent PR I mark Hooks as Thin and Thick wrapper, so we could easily find them in "Some Day"

grep -rl 'thick wrapper' airflow/providers/amazon/aws/hooks
airflow/providers/amazon/aws/hooks/emr.py
airflow/providers/amazon/aws/hooks/dynamodb.py
airflow/providers/amazon/aws/hooks/batch_client.py
airflow/providers/amazon/aws/hooks/kinesis.py
airflow/providers/amazon/aws/hooks/glue.py
airflow/providers/amazon/aws/hooks/sagemaker.py
airflow/providers/amazon/aws/hooks/ec2.py
airflow/providers/amazon/aws/hooks/datasync.py
airflow/providers/amazon/aws/hooks/athena.py
airflow/providers/amazon/aws/hooks/s3.py
airflow/providers/amazon/aws/hooks/elasticache_replication_group.py

Copy link
Contributor

@ferruzzi ferruzzi Jan 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tack onto the "Some Day" list, converting as many hooks as possible to thin hooks, maybe with a Protocol class for IDE type hinting and completion. I didn't like the EcsProtocol and BatchProtocol classes at first, but they grew on me.

Anyway, we should maybe get a list of "easy but tedious suggestions" like this for new contributors to flip through, but we're getting way off track here 😛

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also like an idea @o-nikolas generate hooks for all supported services #28560 (comment)

Copy link
Contributor

@ferruzzi ferruzzi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One discussion left about whether we should be passing region to every hook, but I think that can be sorted in another PR if we do it.

@vandonr-amz
Copy link
Contributor Author

One discussion left about whether we should be passing region to every hook, but I think that can be sorted in another PR if we do it.

yes I'd rather keep this PR well rounded like this

@Taragolis
Copy link
Contributor

One discussion left about whether we should be passing region to every hook, but I think that can be sorted in another PR if we do it.

I think ideally we should pass every generic hook's arguments from operator.
But let do it in another PR

@o-nikolas o-nikolas merged commit 1ab7ea8 into apache:main Jan 20, 2023
@vandonr-amz vandonr-amz deleted the vandonr/hook branch January 20, 2023 00:11
maggesssss pushed a commit to maggesssss/airflow that referenced this pull request Jan 21, 2023
…e#29001)

uniformize getting hook through cached property in aws sensors
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:amazon AWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants