Add AthenaSqlHook based on PyAthena

[eladkal]

Body

The problem:

AthenaOperator store query result in S3 and push to xcom the query_id.
To read the result one must find the path in S3 of the query output (using query_id), download the output file, open it and then ask questions on it. This is very very very frusturating.

My query for example is SELECT(1) FROM.... I simply want to count records and compare it to something. I want the record number to be pushed to Xcom. This enables variety of options such as Branching, ShortCircuit and other

We probably can not (don't want) to just add push_query_result flag to AthenaOperator from what I see the API of the hook doesn't allow this easily.

We should see if we can overcome this somehow. (maybe a new hook with PyAthena? by doing so Athena should be compatible with all our current SQL operators.)
This is a big blocker from utalizing Athena efficiently.

The Task: (at least how I see it)
Create AthenaSqlHook which is based on PyAthena. Hook should inherit from DbApiHook

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[treyyi]

Hi @eladkal,
I would like to take a look and work on this new AthenaSqlHook implementation.
Could you assign this task to me?

Thanks!

[flolas]

Hi @treyyi Are you working on this? I'm really interested in this hook. Happy to help if you need so