DCV-3417 document airflow upload/download + dbt retry (#59)

BAntonellini · web-flow · commit 22130601b04b · 2025-06-22T15:08:20.000-07:00
* DCV-3417 document airflow upload/download + dbt retry
* Add dbt-coves setup docs
diff --git a/docs/_sidebar.md b/docs/_sidebar.md
@@ -4,6 +4,7 @@
   - [Account Pre-reqs](/getting-started/Admin/create-account.md)
   - [Configure Airflow](/getting-started/Admin/configure-airflow.md)
   - [Configure Git Repository](/getting-started/Admin/configure-repository.md)
+  - [Configure Git Repository Using dbt-coves](/getting-started/Admin/configure-repository-using-dbt-coves.md)
   - [Creating Airflow Dags](/getting-started/Admin/creating-airflow-dags.md)
   - [User Management](/getting-started/Admin/user-management.md)
 - [Developer](/getting-started/developer/)
@@ -19,14 +20,15 @@
     - [Airflow - Sync Internal Airflow database](/how-tos/airflow/sync-database.md)
     - [Airflow - Trigger a DAG using Datasets](how-tos/airflow/api-triggered-dag.md)
     - [DAGs - Add Dag Documentation](/how-tos/airflow/create-dag-level-docs.md)
-    - [DAGs - Calling External Python Scripts](/how-tos/airflow/external-python-dag.md) 
+    - [DAGs - Calling External Python Scripts](/how-tos/airflow/external-python-dag.md)
     - [DAGs - Dynamically Set Schedule](/how-tos/airflow/dynamically-set-schedule.md)
     - [DAGs - Generate DAGs from yml](/how-tos/airflow/generate-dags-from-yml.md)
-    - [DAGs - Get Current Git Branch Name from a DAG Task](/how-tos/airflow/get-current-branch-name.md)    
+    - [DAGs - Get Current Git Branch Name from a DAG Task](/how-tos/airflow/get-current-branch-name.md)
     - [DAGS - Load from S3 to Snowflake](/how-tos/airflow/s3-to-snowflake.md)
     - [DAGs - Run ADF Pipelines](/how-tos/airflow/run-adf-pipeline.md)
     - [DAGs - Run Airbyte sync jobs](/how-tos/airflow/run-airbyte-sync-jobs.md)
     - [DAGs - Run dbt](/how-tos/airflow/run-dbt.md)
+    - [DAGs - Retry dbt jobs](/how-tos/airflow/retry-dbt-tasks.md)
     - [DAGs - Run Databricks Notebooks](/how-tos/airflow/run-databricks-notebook.md)
     - [DAGs - Run Fivetran sync jobs](/how-tos/airflow/run-fivetran-sync-jobs.md)
     - [DAGs - Test DAGs](/how-tos/airflow/test-dags.md)
@@ -38,6 +40,7 @@
     - [Secrets - Datacoves Secrets Manager](/how-tos/airflow/use-datacoves-secrets-manager.md)
     - [Worker - Custom Worker Environment](/how-tos/airflow/customize-worker-environment.md)
     - [Worker - Request Memory and CPU](/how-tos/airflow/request-resources-on-workers.md)
+
   - [Datacoves](/how-tos/datacoves/)
     - [Configure Connection Templates](/how-tos/datacoves/how_to_connection_template.md)
     - [Configure Datacoves Secret](/how-tos/datacoves/how_to_secrets.md)
@@ -129,7 +132,7 @@
     - [Datacoves Operators](/reference/airflow/datacoves-operator.md)
   - [Datacoves](/reference/datacoves/)
     - [Versioning](/reference/datacoves/versioning.md)
-    - [VPC Deployment](/reference/datacoves/vpc-deployment.md) 
+    - [VPC Deployment](/reference/datacoves/vpc-deployment.md)
   - [Metrics & Logs](/reference/metrics-and-logs/)
     - [Grafana](/reference/metrics-and-logs/grafana.md)
   - [Security](/reference/security/)
diff --git a/docs/getting-started/Admin/configure-repository-using-dbt-coves.md b/docs/getting-started/Admin/configure-repository-using-dbt-coves.md
@@ -0,0 +1,96 @@
+# Initial Datacoves Repository Setup
+
+## Introduction
+
+Setting up a new data project requires careful consideration of tools, configurations, and best practices. Datacoves simplifies this process by providing a standardized, yet customizable setup through the `dbt-coves` library. This article explains how to initialize and maintain your Datacoves repository.
+
+## Getting Started with dbt-coves Setup
+
+The `dbt-coves setup` command generates a fully configured project environment tailored to your specific needs. This command creates a repository structure with all necessary components pre-configured according to data engineering best practices.
+
+### Initial Setup Process
+
+dbt-coves comes pre-installed in Datacoves, you only have to run:
+
+```bash
+# Create a new Datacoves repository
+dbt-coves setup
+```
+
+During the setup process, you'll be guided through a series of configuration questions that determine:
+
+- Which data warehouse to use (Snowflake, BigQuery, Redshift, Databricks)
+- Which components to include in your stack (dbt, Airflow, dlt)
+- Project naming conventions
+- Repository structure preferences
+- CI/CD pipeline configurations
+- Testing and documentation settings
+
+>[!NOTE] It is recommended that you commit the answers file in your repo for future updates (see below)
+
+## What Gets Created
+
+The `dbt-coves setup` command generates a comprehensive project structure that includes:
+
+1. **dbt configuration**
+   - Pre-configured dbt project with appropriate adapters
+   - Custom macros tailored to your selected data warehouse
+   - Template generators for consistent model creation
+
+2. **Orchestration tools**
+   - Airflow DAG templates (if selected)
+   - Pipeline configurations
+
+3. **Data loading**
+   - dlt configurations for data ingestion (if selected)
+
+4. **Quality control**
+   - SQLFluff and YAMLlint configurations
+   - dbt test frameworks
+   - CI/CD workflows for GitHub Actions or GitLab CI
+
+5. **Documentation**
+   - README templates
+   - Project structure documentation
+
+## Customizing Your Setup
+
+The setup process is highly flexible, allowing you to:
+
+- Select only the components you need
+- Configure folder structures based on your preferences
+- Set up CI/CD pipelines appropriate for your workflow
+- Include specialized macros for your specific data warehouse
+
+## Updating Your Repository
+
+As your project evolves or as Datacoves releases template improvements, you can update your existing repository:
+
+```bash
+# Update an existing Datacoves repository
+dbt-coves setup --update
+```
+
+The update process:
+- Preserves your custom code and configurations
+- Updates template-managed files with the latest versions
+- Adds any new components you select (it will remove the components you selected at Setup time but didn't select at Update time)
+- Maintains backward compatibility where possible
+
+>[!NOTE] When running an update, you will be prompted for the services you want to setup / update, if you saved the answers file from when you first ran set, your original choices pre-selected. If you unselect one of these, that content will be deleted
+
+## Benefits for Data Teams
+
+This approach to repository setup and maintenance offers several advantages:
+
+1. **Reduced setup time** from days to minutes
+2. **Consistency** across projects and teams
+3. **Built-in best practices** for data modeling and CI/CD
+4. **Easy maintenance** through template updates
+5. **Standardized testing** and quality control
+
+## Conclusion
+
+The `dbt-coves setup` command streamlines the creation and maintenance of Datacoves repositories by providing a solid foundation that incorporates industry best practices. Whether you're starting a new data project or standardizing existing ones, this approach offers a scalable and maintainable solution for modern data stack implementation.
+
+By leveraging this setup process, data teams can focus on delivering value through data transformations and insights rather than spending time on infrastructure configuration.
diff --git a/docs/how-tos/airflow/retry-dbt-tasks.md b/docs/how-tos/airflow/retry-dbt-tasks.md
@@ -0,0 +1,98 @@
+# Retry a dbt task
+
+## Overview
+
+Retrying failed dbt models is a common workflow requirement when working with data transformations. This guide explains how to implement dbt task retry functionality in Airflow using Datacoves' custom `datacoves_dbt` decorator.
+
+## Prerequisites
+
+- Datacoves version 3.4 or later
+- dbt API feature enabled in your environment (contact support for further assistance)
+
+## How dbt Retries Work
+
+The retry mechanism works by:
+
+1. **Capturing results** of a dbt run including any failures
+2. **Storing these results** using the dbt API
+3. **Retrieving the previous run state** when a retry is initiated
+4. **Selectively running** only the failed models and their downstream dependencies
+
+## Implementing dbt Retries
+
+### Step 1: Configure the `datacoves_dbt` Decorator
+
+When defining your task, enable the necessary parameters for retries:
+
+```python
+@task.datacoves_dbt(
+    connection_id="your_connection",
+    dbt_api_enabled=True,        # Enable dbt API functionality
+    download_run_results=True,   # Allow downloading previous run results
+)
+```
+
+### Step 2: Add Conditional Logic for Retry
+
+Implement logic in your task function to check for existing results and execute the appropriate dbt command:
+
+```python
+@task.datacoves_dbt(
+    connection_id="your_connection",
+    dbt_api_enabled=True,
+    download_run_results=True,
+)
+def dbt_build(expected_files: list = []):
+    if expected_files:
+        return "dbt build -s result:error+ --state logs"
+    else:
+        return "dbt build -s your_models+"
+```
+
+### Step 3: Call the Task with Expected Files Parameter
+
+```python
+dbt_build(expected_files=["run_results.json"])
+```
+
+## Complete Example
+
+Here's a complete DAG implementation:
+
+```python
+"""
+## Retry dbt Example
+This DAG demonstrates how to retry a DAG that fails during a run
+"""
+
+from airflow.decorators import dag, task
+from orchestrate.utils import datacoves_utils
+
+
+@dag(
+    doc_md = __doc__,
+    catchup = False,
+    default_args=datacoves_utils.set_default_args(
+        owner = "Your Name",
+        owner_email = "your.email@example.com"
+    ),
+    schedule = datacoves_utils.set_schedule("0 0 1 */12 *"),
+    description="Sample DAG demonstrating how to run the dbt models that fail",
+    tags=["dbt_retry"],
+)
+def retry_dbt_failures():
+    @task.datacoves_dbt(
+        connection_id="your_connection",
+        dbt_api_enabled=True,
+        download_run_results=True,
+    )
+    def dbt_build(expected_files: list = []):
+        if expected_files:
+            return "dbt build -s result:error+ --state logs"
+        else:
+            return "dbt build -s model_a+ model_b+"
+
+    dbt_build(expected_files=["run_results.json"])
+
+retry_dbt_failures()
+```
diff --git a/docs/reference/airflow/datacoves-decorators.md b/docs/reference/airflow/datacoves-decorators.md
@@ -27,10 +27,10 @@ def my_bash_dag():
     @task.datacoves_bash
     def echo_hello_world() -> str:
         return "Hello World!"
-
 dag = my_bash_dag()
 ```
 
+
 ### @task.datacoves_dbt
 
 This custom decorator is an extension of the @task decorator and simplifies running dbt commands within Airflow. 
@@ -43,7 +43,7 @@ This custom decorator is an extension of the @task decorator and simplifies runn
 - It runs dbt commands inside the dbt Project Root, not the Repository root.
 
 **Params:**
-- `connection_id`: This is the [service connection](/how-tos/datacoves/how_to_service_connections.md) which is automatically added to airflow if you select `Airflow Connection` as the `Delivery Mode`. 
+- `connection_id`: This is the [service connection](/how-tos/datacoves/how_to_service_connections.md) which is automatically added to airflow if you select `Airflow Connection` as the `Delivery Mode`.
 - `overrides`: Pass in a dictionary with override parameters such as warehouse, role, or database.
 
 ```python
@@ -58,6 +58,7 @@ dag = my_dbt_dag()
 ```
 
 Example with overrides.
+
 ```python
 def my_dbt_dag():
     @task.datacoves_dbt(
@@ -72,6 +73,39 @@ dag = my_dbt_dag()
 The examples above use the Airflow connection `main` which is added automatically from the Datacoves Service Connection
 ![Service Connection](assets/service_connection_main.jpg)
 
+#### Uploading and downloading dbt results
+
+From Datacoves 3.4 onwards, the `datacoves_dbt` decorator allows users to upload and download dbt execution results and metadata to our `dbt API`
+
+>[!NOTE] dbt-API is a feature that is not enabled by default. Please contact support for further assistance.
+
+This is particularly useful for performing [dbt retries](/how-tos/airflow/retry-dbt-tasks.md).
+
+
+The new datacoves_dbt parameters are:
+
+- `dbt_api_enabled` (Default: `False`): Whether your Environment includes a dbt API instance.
+- `download_static_artifacts` (Default: `True`): Whether user wants to download dbt static artifact files.
+- `upload_static_artifacts` (Default: `False`): Whether user wants to upload dbt static files.
+- `download_additional_files` (Default: `[]`): A list of extra paths the user wants to download.
+- `upload_additional_files` (Default: `[]`): A list of extra paths the user wants to upload.
+- `upload_tag` (Default: DAG `run_id`): The tag/label the files will be uploaded with.
+- `upload_run_results` (Default: `True`): Whether the `run_results.json` dbt file will be uploaded.
+- `download_run_results` (Default: `False`): Whether the `run_results.json` dbt file will be downloaded.
+- `upload_sources_json` (Default: `True`): Whether the `sources.json` dbt file will be uploaded.
+- `download_sources_json` (Default: `False`): Whether the `sources.json` dbt file will be downloaded.
+
+>[!NOTE]
+>**Static Artifacts**  
+>The static artifacts are important dbt-generated files that help with dbt's operations:
+>
+>- `target/graph_summary.json`: Contains a summary of the DAG structure of your dbt project.
+>- `target/graph.gpickle`: A serialized Python networkx graph object representing your dbt project's dependency graph.
+>- `target/partial_parse.msgpack`: Used by dbt to speed up subsequent runs by storing parsed information.
+>- `target/semantic_manifest.json`: Contains semantic information about your dbt project.
+>
+>These files are downloaded by default (when `download_static_artifacts=True`) and are tagged as "latest" when uploaded.
+
 ### @task.datacoves_airflow_db_sync
 
 >[!NOTE] The following Airflow tables are synced by default: ab_permission, ab_role, ab_user, dag, dag_run, dag_tag, import_error, job, task_fail, task_instance. 
