DAG with AthenaOperator not producing OpenLineage DAG terminal event #40598
Labels
area:providers
kind:bug
This is a clearly a bug
provider:amazon-aws
AWS/Amazon - related issues
provider:openlineage
AIP-53
Apache Airflow Provider(s)
amazon, openlineage
Versions of Apache Airflow Providers
Tested on both: main branch and latest pypi versions:
apache-airflow-providers-openlineage==1.8.0
apache-airflow-providers-amazon==8.25.0
Apache Airflow version
main branch and 2.9.2
Operating System
MacOS 14.5
Deployment
Astronomer
Deployment details
astro-runtime:11.6.0 (tested with astro dev and actual deployment, they should be the same but still verified it)
Also tested with breeze on main branch
What happened
When running simple DAG with AthenaOperator, I am sometimes not receiving OpenLineage DAG complete events. It looks flaky at first, and I've not yet figured it out. There is nothing suspicious in scheduler logs for me.
I'm not sure how the choice of the task (AthenaOperator) can influence lack of DAG complete event that is emitted from the scheduler, so it's possible that I did something wrong here. Let me know if you are able to reproduce this behaviour.
What you think should happen instead
We should always receive OpenLineage DAG complete events.
How to reproduce
I've run this DAG a couple times, and did not receive DAG complete events.
On astro, this is my dockerfile:
Example DAG
Anything else
Scheduler logs from astro dev:
astro_scheduler_logs.txt
Scheduler and task logs from breeze:
breeze_scheduler_logs.txt
breeze_task_logs.log
Marquez events:
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: