Skip to content

Add remaining steps to the Data refresh DAGs #4149

Closed

Description

Problem

Blocked by #4148

This issue tracks adding the final steps of the data refresh to the new DAGs.

Description

The final steps will be added to the DAGs:

  • Create and Populated Filtered Index: this should be implemented as a reusable TaskGroup. This work can already be implemented using our existing Elasticsearch tasks and replaces this function.
  • Reapply Constraints and Promote Table: these SQL queries can be performed with the PostgresHook, and replaces this function.
  • Promote Index and Delete Index: these can be implemented using our existing Elasticsearch tasks, and replaces these functions.

Additional context

See this section of the IP.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

Labels

⛔ status: blockedBlocked & therefore, not ready for work✨ goal: improvementImprovement to an existing user-facing feature💻 aspect: codeConcerns the software code in the repository🟨 priority: mediumNot blocking but should be addressed soon🧱 stack: catalogRelated to the catalog and Airflow DAGs

Type

No type

Projects

  • Status

    ✅ Done

Relationships

None yet

Development

No branches or pull requests

Issue actions