Dependency is getting lost when the models is getting referred through macros

[paritoshg2010]

Hi Team,

I noticed that dependency graph is getting lost when the models are referred through macros.
For example, in the below screenshot, the model B & C is created from macro where model B's macro is referring model A & model C's macro is referring model B in it. So ideally model A should run first & then B and then C. Here all three are running in parallel when using the DbtTaskGroup. Just to add, this dependency runs as expected when using simple dbt run command.

Through DbtTaskGroup:

Expected:
Model A -> Model B -> Model C

Not sure if this is a known issue, but if you can please check and advise.

[dosubot]

🤖

Hello @paritoshg2010,

Nice to meet you! I'm Dosu, an AI bot here to assist with bugs, answer questions, and guide you on becoming a contributor to the Cosmos repository. While we wait for a human maintainer to join us, I'm here to assist you. Let's work on this together!

The issue you're experiencing is likely due to the fact that Cosmos, when rendering dbt projects as Airflow TaskGroups, does not automatically infer dependencies between models. In your case, the DbtTaskGroup for Model B and Model C are created independently and are not aware of each other. This is different from a simple dbt run command, which automatically builds a dependency graph based on the references in your dbt models.

To maintain the correct order of operations (Model A -> Model B -> Model C) when models are referred through macros, you need to explicitly define the dependencies in your Airflow DAG. This can be done by setting up the dependencies between the DbtTaskGroup instances in your DAG.

Here's an example of how you can do this:

pre_dbt = EmptyOperator(task_id="pre_dbt")

model_a = DbtTaskGroup(
    # ...
)

model_b = DbtTaskGroup(
    # ...
)

model_c = DbtTaskGroup(
    # ...
)

post_dbt = EmptyOperator(task_id="post_dbt")

# Define dependencies
pre_dbt >> model_a >> model_b >> model_c >> post_dbt

In this example, the model_b task group will not start until the model_a task group has successfully completed, and model_c will not start until model_b has completed. This ensures that the models are run in the correct order.

You can refer to the basic_cosmos_task_group.py file in the Cosmos repository for more examples on how to use DbtTaskGroup in your Airflow DAGs.

Sources

• dev/dags/basic_cosmos_task_group.py

[paritoshg2010]

Don't think, if this is going to be a scalable approach.

[dosubot]

🤖

Hi, @paritoshg2010,

I'm helping the Cosmos team manage their backlog and am marking this issue as stale. From what I understand, the issue you reported involved a loss of dependency graph when models are referred through macros, causing DbtTaskGroup instances to run independently instead of sequentially. It looks like the issue has been resolved by implementing a custom solution to maintain the correct order of operations, explicitly defining the dependencies between the instances in the Airflow DAG.

Could you please confirm if this issue is still relevant to the latest version of the Cosmos repository? If it is, please let the Cosmos team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you!

[zzozfe]

Hi @paritoshg2010 ,

I recently encountered the same issue. Have you resolved it? Thanks.