Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate metropolitan reingestion workflow failures/time outs #2920

Open
stacimc opened this issue Aug 30, 2023 · 0 comments
Open

Investigate metropolitan reingestion workflow failures/time outs #2920

stacimc opened this issue Aug 30, 2023 · 0 comments
Labels
💻 aspect: code Concerns the software code in the repository 🛠 goal: fix Bug fix 🟩 priority: low Low priority and doesn't need to be rushed 🧱 stack: catalog Related to the catalog and Airflow DAGs

Comments

@stacimc
Copy link
Contributor

stacimc commented Aug 30, 2023

Description

The Metropolitan reingestion DAG take an extremely long time. It is a weekly DAG and historically times out at the end of the week having only completed 6 or 7 reingestion days, and then starts over. This means reingestion is not getting completed on the proper schedule, and the DAG is perpetually running (so it is almost always stopped and forced to error when a deploy runs).

We need to investigate why the DAG takes such a long time and consider:

  • are there more efficient ways to reingest?
  • can we alter the reingestion schedule?
  • can we make the DAG resilient to being shut down during a deploy (similar to how batched updates work)?

This issue is for investigating the problem. We may need to open new issues for work, depending on our findings.

Additional context

@stacimc stacimc added 🟩 priority: low Low priority and doesn't need to be rushed 🛠 goal: fix Bug fix 💻 aspect: code Concerns the software code in the repository 🧱 stack: catalog Related to the catalog and Airflow DAGs labels Aug 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
💻 aspect: code Concerns the software code in the repository 🛠 goal: fix Bug fix 🟩 priority: low Low priority and doesn't need to be rushed 🧱 stack: catalog Related to the catalog and Airflow DAGs
Projects
Status: 📋 Backlog
Development

No branches or pull requests

1 participant