-
Notifications
You must be signed in to change notification settings - Fork 16.4k
uniformize getting hook through cached property in aws sensors #29001
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some of these files you are passing region to the hook and others are not. Should we standardize that as well?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I mostly kept the existing behavior. I don't know what is a real use case for passing the region, maybe that depends on the operator/sensor ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Somewhere in the past I have an idea to keep Sensors/Operators more consistent and provide all of supported parameters in Hook.
I think it safe to pass region_name because we set it to None by default in Operators/Sensors.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
BTW , it is not related to this PR, but we have a 3 different hooks which is wraps around Glue.Client 🤦
- airflow.providers.amazon.aws.hooks.glue.GlueJobHook
- airflow.providers.amazon.aws.hooks.glue_catalog.GlueCatalogHook
- airflow.providers.amazon.aws.hooks.glue_crawler.GlueCrawlerHook
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
BTW , it is not related to this PR, but we have a 3 different hooks which is wraps around Glue.Client facepalm
Yeah, there is also a BatchWaiterHook which is an oddball too. Some Day ™️
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In one recent PR I mark Hooks as Thin and Thick wrapper, so we could easily find them in "Some Day"
❯ grep -rl 'thick wrapper' airflow/providers/amazon/aws/hooks
airflow/providers/amazon/aws/hooks/emr.py
airflow/providers/amazon/aws/hooks/dynamodb.py
airflow/providers/amazon/aws/hooks/batch_client.py
airflow/providers/amazon/aws/hooks/kinesis.py
airflow/providers/amazon/aws/hooks/glue.py
airflow/providers/amazon/aws/hooks/sagemaker.py
airflow/providers/amazon/aws/hooks/ec2.py
airflow/providers/amazon/aws/hooks/datasync.py
airflow/providers/amazon/aws/hooks/athena.py
airflow/providers/amazon/aws/hooks/s3.py
airflow/providers/amazon/aws/hooks/elasticache_replication_group.py
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tack onto the "Some Day" list, converting as many hooks as possible to thin hooks, maybe with a Protocol class for IDE type hinting and completion. I didn't like the EcsProtocol and BatchProtocol classes at first, but they grew on me.
Anyway, we should maybe get a list of "easy but tedious suggestions" like this for new contributors to flip through, but we're getting way off track here 😛
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I also like an idea @o-nikolas generate hooks for all supported services #28560 (comment)
ferruzzi
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One discussion left about whether we should be passing region to every hook, but I think that can be sorted in another PR if we do it.
yes I'd rather keep this PR well rounded like this |
412603b to
b26d970
Compare
I think ideally we should pass every generic hook's arguments from operator. |
…e#29001) uniformize getting hook through cached property in aws sensors
also caught some hooks that were created in the ctor.