Skip to content

Make amazon athena, batch, lambda, and eks operators more consistent #24246

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
Taragolis wants to merge 2 commits into from
Closed

Make amazon athena, batch, lambda, and eks operators more consistent #24246

Taragolis wants to merge 2 commits into from

Conversation

@Taragolis
Copy link
Contributor

@Taragolis Taragolis commented Jun 6, 2022 •
edited
Loading

related: #24198

Try to make amazon-provider operators (and later sensors) more consistent.
At that moment I try to use Generic and additional MixIn class for passing parameters to hooks and simplify operators itself

Affected different types of operators

  • airflow.airflow.providers.amazon.aws.operators.athena.AthenaOperator
  • airflow.airflow.providers.amazon.aws.operators.aws_lambda.AwsLambdaInvokeFunctionOperator
  • airflow.airflow.providers.amazon.aws.operators.batch.BatchOperator
  • airflow.airflow.providers.amazon.aws.operators.eks.EksCreateClusterOperator
  • airflow.airflow.providers.amazon.aws.operators.eks.EksCreateNodegroupOperator
  • airflow.airflow.providers.amazon.aws.operators.eks.EksCreateFargateProfileOperator
  • airflow.airflow.providers.amazon.aws.operators.eks.EksDeleteClusterOperator
  • airflow.airflow.providers.amazon.aws.operators.eks.EksDeleteNodegroupOperator
  • airflow.airflow.providers.amazon.aws.operators.eks.EksDeleteFargateProfileOperator
  • airflow.airflow.providers.amazon.aws.operators.eks.EksPodOperator

@boring-cyborg boring-cyborg bot added area:providers provider:amazon AWS/Amazon - related issues labels Jun 6, 2022
@Taragolis
Copy link
Contributor Author

Taragolis commented Jun 6, 2022

looks on failed tests 👀

@Taragolis Taragolis force-pushed the makes-amazon-provider-more-consistent branch from 8e203ef to 60b8227 Compare June 6, 2022 14:08
@ferruzzi
Copy link
Contributor

ferruzzi commented Jun 7, 2022

@o-nikolas should have eyes on this one. It implements a couple of things we've discussed.

ferruzzi
ferruzzi reviewed Jun 7, 2022
View reviewed changes
ferruzzi
ferruzzi reviewed Jun 7, 2022
View reviewed changes
uranusjr
uranusjr reviewed Jun 7, 2022
View reviewed changes
uranusjr
uranusjr reviewed Jun 7, 2022
View reviewed changes
@ferruzzi
Copy link
Contributor

ferruzzi commented Jun 7, 2022 •
edited
Loading

looks on failed tests eyes

You just need to add the base operator module to airflow/providers/amazon/provider.yaml around L217

@Taragolis Taragolis force-pushed the makes-amazon-provider-more-consistent branch from 866740c to fc38663 Compare June 8, 2022 09:14
Taragolis
Taragolis commented Jun 8, 2022
View reviewed changes
airflow/providers/amazon/aws/operators/base.py Outdated
Comment on lines 22 to 25
Copy link
Contributor Author

@Taragolis Taragolis Jun 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, since new provider versions expected min Airflow 2.2 could we replace this by

from airflow.compat.functools import cached_property

@Taragolis Taragolis force-pushed the makes-amazon-provider-more-consistent branch from 29ea2bd to e430ab7 Compare June 8, 2022 14:44
@Taragolis Taragolis requested a review from uranusjr June 8, 2022 15:49
@Taragolis
Copy link
Contributor Author

Taragolis commented Jun 8, 2022

Recent changes:

  • aws_hook_class_kwargs class argument is optional
  • check AwsBaseHook.conn_name_attr for non-default aws_conn_id and use it in hook
  • for EKS operator replace region to region_name. Previously created DAGs shouldn't failed
  • Add tests

cc: @ferruzzi @uranusjr @o-nikolas

ferruzzi
ferruzzi reviewed Jun 8, 2022
View reviewed changes
@github-actions
Copy link

github-actions bot commented Jul 25, 2022

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Jul 25, 2022
@Taragolis
Copy link
Contributor Author

Taragolis commented Jul 25, 2022

Close for now, plan back to this PR after changes in Hook(s)

@Taragolis Taragolis closed this Jul 25, 2022
@Taragolis Taragolis deleted the makes-amazon-provider-more-consistent branch January 14, 2023 18:32
@Taragolis Taragolis mentioned this pull request Oct 5, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ferruzzi ferruzzi ferruzzi left review comments

@uranusjr uranusjr Awaiting requested review from uranusjr

Assignees

No one assigned

Labels

area:providers provider:amazon AWS/Amazon - related issues stale Stale PRs per the .github/workflows/stale.yml policy file

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Taragolis @ferruzzi @uranusjr