Add support for `dbt Cloud` as Execution Mode

[jaklan]

Currently astronomer-cosmos supports following execution modes:

Execution Modes

Cosmos can run dbt commands using five different approaches, called execution modes:

1. local: Run dbt commands using a local dbt installation (default)

2. virtualenv: Run dbt commands from Python virtual environments managed by Cosmos

3. docker: Run dbt commands from Docker containers managed by Cosmos (requires a pre-existing Docker image)

4. kubernetes: Run dbt commands from Kubernetes Pods managed by Cosmos (requires a pre-existing Docker image)

5. azure_container_instance: Run dbt commands from Azure Container Instances managed by Cosmos (requires a pre-existing Docker image)

The choice of the execution mode can vary based on each user’s needs and concerns. For more details, check each execution mode described below.

It would be great to see dbt Cloud in that list as well, so we can automatically create and trigger remote jobs using dbt Cloud API under the hood.

[dosubot]

That sounds like a great idea! Adding support for dbt Cloud as an Execution Mode would definitely enhance the functionality of the astronomer-cosmos project. If you're interested in contributing this feature, feel free to create a pull request with your proposed changes. Your contribution would be much appreciated!

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[tatiana]

@jaklan, this would be a nice feature, especially when leveraging deferrable operators. Would you like to contribute it?

[jaklan]

We don't have resources for that, I would propose to reach to guys responsible for dbt Cloud Airflow provider: https://github.com/apache/airflow/tree/main/airflow/providers/dbt/cloud

[tatiana]

@jaklan I agree this is a relevant feature, but we also have limited bandwidth. If someone from the community is keen on implementing this, we can support them.

[dosubot]

Hi, @jaklan. I'm Dosu, and I'm helping the Cosmos team manage their backlog. I'm marking this issue as stale.

Issue Summary:

• You suggested adding support for dbt Cloud as an execution mode in astronomer-cosmos.
• The feature would enable automatic creation and triggering of remote jobs using the dbt Cloud API.
• Both Tatiana and I expressed support, highlighting benefits with deferrable operators.
• You mentioned a lack of resources to implement the feature and suggested collaboration with the dbt Cloud Airflow provider team.
• Tatiana acknowledged the feature's importance but noted limited bandwidth, welcoming community contributions.

Next Steps:

• Please let us know if this issue is still relevant to the latest version of the Cosmos repository by commenting here.
• If there is no further activity, this issue will be automatically closed in 2400 days.

Thank you for your understanding and contribution!

[jaklan]

Relevant