datadog: Support logging of full metrics payloads

[ringerc]

Component(s)

exporter/datadog

Is your feature request related to a problem? Please describe.

The datadog exporter has the logs.dump_payloads configuration option, but it does not appear to have any similar one for metrics (or traces).

The datadog exporter does its own internal munging of metrics, transforming resource attributes to various datadog tags, etc. This is affected by lightly documented config options like resource_attributes_as_tags, but it can be hard to tell what exactly the DD exporter is doing to the metrics before delivery.

Since the DD ingestion pipeline also has various configuration for tag remapping (some of it present by default) it can be hard to tell where tags like "cloud_provider" or "zone" come from. They don't appear in the "debug" exporter stream when set to log the detailed payloads, so they're being added by Datadog either on the collector or on the backend... but where, why, and how?

Describe the solution you'd like

A metrics.dump_payloads (and traces.dump_payloads) option to log the full payloads as they will be sent to the Datadog API endpoint.

Describe alternatives you've considered

Enabling debug log level for the exporter as a whole does not appear to cause the datadog exporter to log payload details. Which is fine, that's probably not desirable, but it'd be good to have some way to do this.

Additional context

See #29702 for added confusion the apparently-ignored metrics.resource_attributes_as_tags option introduces, and the undocumented mapping of OpenTelemetry semantic conventions resource attributes the Datadog exporter does to datadog tags.

[github-actions]

Pinging code owners:

• exporter/datadog: @mx-psi @gbbr @dineshg13 @liustanley @songy23 @mackjmr

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[ringerc]

This would've made it way easier to figure out the behaviour in #29702 for example.

It's especially important since the Datadog platform offers no way to see the "raw" feed of incoming metrics with each datum's tags etc. You can only see the summary, which combines all recent tags from all sources. Fiddly queries are required to figure out what's actually being sent currently by any given source.

[mackjmr]

Hi @ringerc, I've created a task in our backlog to add support for dump_payloads for traces/metrics. For tracking purposes, can you please open a support ticket referencing this issue ?

[ringerc]

@mackjmr Done, DD support ticket 1486970