Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove the recreate_full_staging_index DAG #4152

Open
Tracked by #3925
stacimc opened this issue Apr 17, 2024 · 0 comments
Open
Tracked by #3925

Remove the recreate_full_staging_index DAG #4152

stacimc opened this issue Apr 17, 2024 · 0 comments
Labels
💻 aspect: code Concerns the software code in the repository ✨ goal: improvement Improvement to an existing user-facing feature 🟨 priority: medium Not blocking but should be addressed soon 🧱 stack: catalog Related to the catalog and Airflow DAGs ⛔ status: blocked Blocked & therefore, not ready for work
Milestone
Removal of the da...

Comments

@stacimc
Copy link
Contributor

stacimc commented Apr 17, 2024

Current Situation

Blocked by #4150

The recreate_full_staging_index DAG performs the second half of a data refresh (the distributed reindex and promotion) in staging. It uses the ingestion server to do so.

Suggested Improvement

Once the data refresh has been moved to Airflow, we can add a DAG param to the data refresh that allows skipping the initial Copy Data steps. The recreate_full_staging_index DAG will no longer be necessary and can then be deleted.

Additional context

See this section of the IP
@stacimc stacimc added 🟨 priority: medium Not blocking but should be addressed soon ✨ goal: improvement Improvement to an existing user-facing feature 💻 aspect: code Concerns the software code in the repository ⛔ status: blocked Blocked & therefore, not ready for work 🧱 stack: catalog Related to the catalog and Airflow DAGs labels Apr 17, 2024
This was referenced Apr 17, 2024
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
💻 aspect: code Concerns the software code in the repository ✨ goal: improvement Improvement to an existing user-facing feature 🟨 priority: medium Not blocking but should be addressed soon 🧱 stack: catalog Related to the catalog and Airflow DAGs ⛔ status: blocked Blocked & therefore, not ready for work
Projects
Openverse Backlog
Status: ⛔ Blocked
Milestone
Removal of the data refresh server
Development

No branches or pull requests
1 participant
@stacimc