Skip to content

Conversation

@wonhyeongseo
Copy link
Contributor

closes: #29091

Simple fix of type annotation for num_of_dpus according to AWS Glue Documentation

@Taragolis may you please review this PR?

@boring-cyborg boring-cyborg bot added area:providers provider:amazon AWS/Amazon - related issues labels Jan 26, 2023
@Taragolis
Copy link
Contributor

You also need to change in Operator (potentially in other places)

num_of_dpus: int | None = None,

I still don't know what is better annotation for this case float | None or int | float | None

@Taragolis
Copy link
Contributor

Yeah, statics check failed:

airflow/providers/amazon/aws/hooks/glue.py:97: error: Incompatible types in
assignment (expression has type "float", variable has type "int")  [assignment]
                self.num_of_dpus = num_of_dpus

@wonhyeongseo wonhyeongseo force-pushed the gluejobhook-type-annotation branch from 5226f3d to c4c14c0 Compare January 26, 2023 11:59
@wonhyeongseo
Copy link
Contributor Author

Ok @Taragolis. Adding in float instead of replacing int for backward compatability. I checked the files for num_of_dpus, and the modified two files are all there is. May you please run the workflow again? Thanks!

Copy link
Contributor

@Taragolis Taragolis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pending approval (wait for CI)

@Taragolis
Copy link
Contributor

🤣 mypy became crazy when tried to make static check against condition in Hook

https://github.com/apache/airflow/blob/c4c14c07a99b1554346a426aa947e68eb52ce31e/airflow/providers/amazon/aws/hooks/glue.py#L95

Add annotation here, something like: self.num_of_dpus: int | float = 10

@wonhyeongseo wonhyeongseo force-pushed the gluejobhook-type-annotation branch from c4c14c0 to f52a89a Compare January 27, 2023 02:28
@wonhyeongseo wonhyeongseo force-pushed the gluejobhook-type-annotation branch from 7c2b962 to 5437a85 Compare January 27, 2023 03:11
@wonhyeongseo
Copy link
Contributor Author

image
Updated as you suggested, @Taragolis! Thank you so much for guiding me. I hope to contribute more in the future. Have a great day/afternoon!

@potiuk potiuk merged commit 4402456 into apache:main Jan 27, 2023
@boring-cyborg
Copy link

boring-cyborg bot commented Jan 27, 2023

Awesome work, congrats on your first merged pull request!

@wonhyeongseo wonhyeongseo deleted the gluejobhook-type-annotation branch January 27, 2023 11:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:amazon AWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Incorrect type annotation for num_of_dpus in GlueJobOperator/GlueJobHook

3 participants