[DNAINT-1516][DJM] Add `dbt` documentation #26994

MassyB · 2025-01-07T14:53:44Z

What does this PR do? What is the motivation?

DJM aims at supporting dbt pipelines. This PR updates our public documentation to allow our customer to send us dbt OpenLineage events.

I took inspiration from this #25196

Jira https://datadoghq.atlassian.net/browse/DNAINT-1516

Merge instructions

[ ] Please merge after reviewing

github-actions · 2025-01-07T14:56:44Z

Preview links (active after the `build_preview` check completes)

New or renamed files

https://docs-staging.datadoghq.com/massy.bourennani/dnaint-1516-dbt-ol-documentation/data_jobs/dbt

Modified Files

https://docs-staging.datadoghq.com/massy.bourennani/dnaint-1516-dbt-ol-documentation/data_jobs/

yohannj · 2025-01-07T16:05:46Z

content/en/data_jobs/dbt.md

+
+**Optional**
+
+1. Setup `OPENLINEAGE_DBT_LOGGING` environment variable, you can establish the logging level for the `openlineage.dbt` and its child modules.


Suggested change

1. Setup `OPENLINEAGE_DBT_LOGGING` environment variable, you can establish the logging level for the `openlineage.dbt` and its child modules.

1. Setup `OPENLINEAGE_DBT_LOGGING` environment variable, you can establish the logging level for the `openlineage.dbt` and its child modules.

I do not fully understand this sentence.
I suppose it's "By setting up ..."
Also by "establish", do you mean I can choose the logging level that way?

Would be great to include an example and briefly explain what the example does.

Wondering if it might be better to put it in the Troubleshooting section, like adding OPENLINEAGE_DBT_LOGGING=DEBUG to enable debug logging

Yep I agree. Putting it in the Troubleshooting section makes sense

content/en/data_jobs/dbt.md

paul-laffon-dd · 2025-01-08T13:14:34Z

content/en/data_jobs/dbt.md

+
+**Optional**
+
+1. Setup `OPENLINEAGE_DBT_LOGGING` environment variable, you can establish the logging level for the `openlineage.dbt` and its child modules.


Wondering if it might be better to put it in the Troubleshooting section, like adding OPENLINEAGE_DBT_LOGGING=DEBUG to enable debug logging

paul-laffon-dd · 2025-01-08T13:16:15Z

content/en/data_jobs/dbt.md

+
+In Datadog, you can see the traces by using the following APM query:
+```text
+operation_name:*dbt*


@warrierr what do you think of pointing users directly to traces for now ?

I don't want to ship this to customers yet without some minimal experience in the DJM UI. I think this sets an odd/inconsistent standard for customers so we should wait until the overview/details page is minimally functional for dbt.

aboitreaud · 2025-01-08T15:46:58Z

layouts/partials/data_jobs/setup-platforms.html

I believe that updating this file to the content below should make two rows of three icons of equal height each:

{{ $dot := . }} <div class="dsm-containers"> <div class="container cards-dd"> <div class="row row-cols-1 row-cols-md-3 g-2 g-xl-3 justify-content-sm-center"> <div class="col"> <a class="card h-100" href="/data_jobs/emr"> <div class="card-body text-center py-2 px-1"> {{ partial "img.html" (dict "root" . "src" "integrations_logos/amazon_emr.png" "class" "img-fluid" "alt" "Amazon EMR" "width" "200") }} </div> </a> </div> <div class="col"> <a class="card h-100" href="/data_jobs/databricks/"> <div class="card-body text-center py-2 px-1"> {{ partial "img.html" (dict "root" . "src" "integrations_logos/databricks.png" "class" "img-fluid" "alt" "Databricks" "width" "200") }} </div> </a> </div> <div class="col"> <a class="card h-100" href="/data_jobs/dataproc/"> <div class="card-body text-center py-2 px-1"> {{ partial "img.html" (dict "root" . "src" "integrations_logos/google_cloud_dataproc.png" "class" "img-fluid" "alt" "GCP Dataproc" "width" "200") }} </div> </a> </div> <div class="col"> <a class="card h-100" href="/data_jobs/kubernetes/"> <div class="card-body text-center py-2 px-1"> {{ partial "img.html" (dict "root" . "src" "integrations_logos/kubernetes.png" "class" "img-fluid" "alt" "Kubernetes" "width" "200") }} </div> </a> </div> <div class="col"> <a class="card h-100" href="/data_jobs/airflow/"> <div class="card-body text-center py-2 px-1"> {{ partial "img.html" (dict "root" . "src" "integrations_logos/airflow.png" "class" "img-fluid" "alt" "Airflow" "width" "200") }} </div> </a> </div> <div class="col"> <a class="card h-100" href="/data_jobs/dbt/"> <div class="card-body text-center py-2 px-1"> {{ partial "img.html" (dict "root" . "src" "integrations_logos/dbt.png" "class" "img-fluid" "alt" "dbt" "width" "140") }} </div> </a> </div> </div> </div> </div>

Co-authored-by: Michael Cretzman <58786311+michaelcretzman@users.noreply.github.com>

warrierr · 2025-01-08T16:42:32Z

content/en/data_jobs/dbt.md

+      tag: 'Documentation'
+      text: 'Data Jobs Monitoring'
+---
+


We need to add a "preview" callout like we do for Airflow

warrierr · 2025-01-08T16:44:40Z

content/en/data_jobs/dbt.md

+
+  * Replace `DD_DATA_OBSERVABILITY_INTAKE` with `https://data-obs-intake.`{{< region-param key="dd_site" code="true" >}}.
+  * Replace `DD_API_KEY` with your valid [Datadog API key][5].
+  * Replace `NAMESPACE` if you want to use something other than the `default` namespace for job namespace.


@MassyB @paul-laffon-dd how should users think of namespace in the context of dbt? What "thing" in dbt do we recommend they name this?

One way is to use different namespaces for different envs (prod/dev/staging)

warrierr · 2025-01-08T16:49:52Z

content/en/data_jobs/dbt.md

+## Validation
+
+In your setup, you can run the following `dbt-ol` command to see traces in Datadog.
+For example, if you are using the [jaffle-shop][8] project:


Suggested change

For example, if you are using the [jaffle-shop][8] project:

I don't think we need to link to this example. I would just make the command generic to a project name to something like this

dbt-ol run --select <your_project_name>

dbt-ol run --select <your_model_name>

warrierr · 2025-01-08T16:53:12Z

content/en/data_jobs/dbt.md

+```
+
+The above consumes dbt [artifacts][9] and sends OpenLineage events **after** the job finishes.
+If you want to receive events in realtime you can use the `--consume-structured-logs` of `dbt-ol`.


Why are we making this an option for customers, shouldn't we just have this option set in the command we tell customers to run? What's the downside of having them get the data in realtime?

What's the downside of having them get the data in realtime?

None. I'll only keep this option then.

warrierr · 2025-01-08T16:55:11Z

content/en/data_jobs/dbt.md

+dbt-ol --consume-structured-logs run --select orders
+```
+
+In Datadog, you can see the traces by using the following APM query:


Everything above here isn't really validation, it is part of the main setup to be able to get the runs to show up in Datadog for your projects. Validation section should purely be where in the DD product to look to see if data is collected correctly.

warrierr · 2025-01-08T16:57:30Z

content/en/data_jobs/dbt.md

+
+## Validation
+
+In your setup, you can run the following `dbt-ol` command to see traces in Datadog.


We should provide a setence about how dbt-ol is a wrapper around dbt and, it supports all of the standard dbt subcommands, and is safe to use as a substitution.

warrierr · 2025-01-08T16:58:46Z

content/en/data_jobs/dbt.md

+
+In Datadog, you can see the traces by using the following APM query:
+```text
+operation_name:*dbt*


I don't want to ship this to customers yet without some minimal experience in the DJM UI. I think this sets an odd/inconsistent standard for customers so we should wait until the overview/details page is minimally functional for dbt.

MassyB marked this pull request as ready for review January 7, 2025 15:16

MassyB requested a review from a team as a code owner January 7, 2025 15:16

yohannj reviewed Jan 7, 2025

View reviewed changes

michaelcretzman requested changes Jan 7, 2025

View reviewed changes

MassyB requested a review from a team as a code owner January 8, 2025 13:14

github-actions bot added the Architecture Everything related to the Doc backend label Jan 8, 2025

paul-laffon-dd reviewed Jan 8, 2025

View reviewed changes

github-actions bot added the Images Images are added/removed with this PR label Jan 8, 2025

aboitreaud reviewed Jan 8, 2025

View reviewed changes

MassyB and others added 9 commits January 8, 2025 16:53

add dbt ol documentation

cf00e0d

vale

4091168

add dbt icone

806a8c4

eof

1750b35

Apply suggestions from code review

77b80e7

Co-authored-by: Michael Cretzman <58786311+michaelcretzman@users.noreply.github.com>

eof

bfd3208

add icone; update Troubleshooting section

fb3f6a4

index layout

bf589a5

replace png

7ba62e5

warrierr requested changes Jan 8, 2025

View reviewed changes

michaelcretzman approved these changes Jan 8, 2025

View reviewed changes

refactor setup and validation

db587b8

MassyB force-pushed the massy.bourennani/dnaint-1516-dbt-ol-documentation branch from 142dcab to db587b8 Compare January 9, 2025 10:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DNAINT-1516][DJM] Add `dbt` documentation #26994

[DNAINT-1516][DJM] Add `dbt` documentation #26994

MassyB commented Jan 7, 2025 •

edited

Loading

github-actions bot commented Jan 7, 2025 •

edited

Loading

yohannj Jan 7, 2025

paul-laffon-dd Jan 8, 2025

MassyB Jan 8, 2025

paul-laffon-dd Jan 8, 2025

paul-laffon-dd Jan 8, 2025

warrierr Jan 8, 2025

aboitreaud Jan 8, 2025

warrierr Jan 8, 2025

warrierr Jan 8, 2025

MassyB Jan 9, 2025

warrierr Jan 8, 2025

MassyB Jan 9, 2025 •

edited

Loading

warrierr Jan 8, 2025

MassyB Jan 9, 2025

warrierr Jan 8, 2025

warrierr Jan 8, 2025

warrierr Jan 8, 2025


		Optional

		1. Setup `OPENLINEAGE_DBT_LOGGING` environment variable, you can establish the logging level for the `openlineage.dbt` and its child modules.


		## Validation

		In your setup, you can run the following `dbt-ol` command to see traces in Datadog.

[DNAINT-1516][DJM] Add dbt documentation #26994

Are you sure you want to change the base?

[DNAINT-1516][DJM] Add dbt documentation #26994

Conversation

MassyB commented Jan 7, 2025 • edited Loading

What does this PR do? What is the motivation?

Merge instructions

github-actions bot commented Jan 7, 2025 • edited Loading

Preview links (active after the build_preview check completes)

New or renamed files

Modified Files

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MassyB Jan 9, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

[DNAINT-1516][DJM] Add `dbt` documentation #26994

[DNAINT-1516][DJM] Add `dbt` documentation #26994

MassyB commented Jan 7, 2025 •

edited

Loading

github-actions bot commented Jan 7, 2025 •

edited

Loading

Preview links (active after the `build_preview` check completes)

MassyB Jan 9, 2025 •

edited

Loading