Closed
Description
Problem
We need a new DAG to be able to trigger the creation of the filtered indexes.
To ease local testing, this is blocked by #1196.
Description
Please see the technical description from the implementation plan here: https://docs.openverse.org/projects/proposals/detecting_sensitive_textual_content/20230308-implementation_plan_filtering_and_designating_results_with_sensitive_textual_content.html#airflow-data-refresh-dag-factory-overview-step-2
Steps to complete:
- Create a new DAG factory function to create a DAG that completes the steps described in the implementation plan section linked above (search for "It must do the following"). Remember to add the anti-race-condition checks by checking that the data refresh DAG for the media type is not currently active. If it is, the DAG should fail.
- Trigger the new DAG from the data refresh DAG at the end, passing the
force
parameter to skip the anti-race-condition check. Add a new Airflow Variable to act as the feature flag for triggering the filtered index creation DAG only when it is enabled.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Metadata
Assignees
Labels
Type
Projects
Status
✅ Done