Existing Airflow DAG and dbt model cleanup for every data source

[jrlegrand]

Problem Statement

We need a wholesale review of our current Airflow DAGs and dbt setup for each data source from beginning to end.

Criteria for Success

Every checkbox in the DAGs and Work section of our Roadmap is checked and complete.

Airflow Standardized

• Abstraction
  • Use airflow operator to create dag and tasks
  • Common tasks used to centralize logic
• Loaded
  • Sources loaded to data lake

Dbt Buildout

• Transformations in staging models
• Combinations in intermediate models
• Final table in prod schema mart model

Data Documented

• Source data documented
• Data source reference linked
• Any additional tables documented

Additional Information

Roadmap: https://docs.google.com/document/d/1LTWPbgTdGrlRq2qi5KAyj68rtyGJfYRB_luUIN_1WFo/edit#heading=h.p5fi4rrf2kkp

Go down to the DAGs and Work section at the bottom.