Skip to content

Create create_filtered_<media_type>_index DAG factory #1197

Closed

Description

Problem

We need a new DAG to be able to trigger the creation of the filtered indexes.

To ease local testing, this is blocked by #1196.

Description

Please see the technical description from the implementation plan here: https://docs.openverse.org/projects/proposals/detecting_sensitive_textual_content/20230308-implementation_plan_filtering_and_designating_results_with_sensitive_textual_content.html#airflow-data-refresh-dag-factory-overview-step-2

Steps to complete:

  1. Create a new DAG factory function to create a DAG that completes the steps described in the implementation plan section linked above (search for "It must do the following"). Remember to add the anti-race-condition checks by checking that the data refresh DAG for the media type is not currently active. If it is, the DAG should fail.
  2. Trigger the new DAG from the data refresh DAG at the end, passing the force parameter to skip the anti-race-condition check. Add a new Airflow Variable to act as the feature flag for triggering the filtered index creation DAG only when it is enabled.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Labels

✨ goal: improvementImprovement to an existing user-facing feature💻 aspect: codeConcerns the software code in the repository🟨 priority: mediumNot blocking but should be addressed soon🧱 stack: catalogRelated to the catalog and Airflow DAGs

Type

No type

Projects

  • Status

    ✅ Done

Relationships

None yet

Development

No branches or pull requests

Issue actions