Investigate metropolitan reingestion workflow failures/time outs #2920
Labels
💻 aspect: code
Concerns the software code in the repository
🛠 goal: fix
Bug fix
🟩 priority: low
Low priority and doesn't need to be rushed
🧱 stack: catalog
Related to the catalog and Airflow DAGs
Description
The Metropolitan reingestion DAG take an extremely long time. It is a weekly DAG and historically times out at the end of the week having only completed 6 or 7 reingestion days, and then starts over. This means reingestion is not getting completed on the proper schedule, and the DAG is perpetually running (so it is almost always stopped and forced to error when a deploy runs).
We need to investigate why the DAG takes such a long time and consider:
This issue is for investigating the problem. We may need to open new issues for work, depending on our findings.
Additional context
The text was updated successfully, but these errors were encountered: