Refactor poll_query_status in EMRContainerHook

[eladkal]

Body

The goal is to refactor the code so that we can remove this TODO

airflow/airflow/providers/amazon/aws/hooks/emr_containers.py

Lines 174 to 176 in 7640ba4

	# TODO: Make this logic a little bit more robust.
	# Currently this polls until the state is *not* one of the INTERMEDIATE_STATES
	# While that should work in most cases...it might not. :)

More information about the concerns can be found on #16766 (comment)

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[patrikhv]

Hi. I would like to work on this issue. Can you assign it to me?

[uranusjr]

Go ahead!

[dacort]

Thanks @patrikhv - let me know if I can help in any way! (original PR contributor and EMR team member)

[patrikhv]

Hi, I am working on that, but I do not have any idea how to refactor it. I already just did a more explicit call, if the query is in a completed stage. Can you give me any tips?

[ferruzzi]

Had a chat with dacort. They mentioned that the change they had in mind was that we don't actually confirm that the state is at a terminal state, only that it is not at an intermediate state. I can't think of any other changes I'd make to it.

[ferruzzi]

@patrikhv - It looks to me like adding in the explicit "is it a terminal state?" check is all that is needed here. You want to put in the PR?

[eladkal]

@ferruzzi since patrikhv is not working on it. Do you want to put a PR to solve this?

[ferruzzi]

@eladkal I have a few other things I'm working on. I can have a look when I get time, but it'll likely be a couple weeks unless somebody gets to it before then.