Datadog exporter not rendering log attributes correctly

[paulius-valiunas]

Component(s)

exporter/datadog

What happened?

Datadog exporter does not export logs in a format that Datadog UI can render correctly

Description

Datadog does not support nested JSON structures in log attributes, so if I have a log message with attributes like:

{
  "foo": "value1",
  "bar": {
    "baz": "some value with \"escaped\" characters",
    "iRanOutOfNames": 2,
    "soIPutSomeGibberishHere": 3
  }
}

foo will be rendered correctly, but the value of bar will appear as a JSON string with all the entries squished into one line, as opposed to being correctly rendered as a tree.

However, if I manually convert the attributes to this (in my application, before sending logs to the collector):

{
  "foo": "value1",
  "bar.baz": "some value with \"escaped\" characters",
  "bar.iRanOutOfNames": 2,
  "bar.soIPutSomeGibberishHere": 3
}

Then bar will be rendered as an expandable tree branch with the children baz, iRanOutOfNames and soIPutSomeGibberishHere.

Steps to Reproduce

1. Have a Datadog exporter
2. Log something with nested attributes (like the first example above)

Expected Result

Datadog shows baz, iRanOutOfNames and soIPutSomeGibberishHere as separate attributes under the parent bar. I should be able to see correctly rendered (unescaped) some value with "escaped" characters next to the property name baz.

Actual Result

Instead, Datadog shows bar as a single attribute with the JSON structure all squished into one line with JSON escape sequences etc.

Collector version

0.70.0

Environment information

Environment

Using opentelemetry-operator Helm chart version 0.24.0

OpenTelemetry Collector configuration

No response

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/datadog: @mx-psi @gbbr @dineshg13 @liustanley @songy23 @mackjmr

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[peter-tar]

I would be interested in a fix for this as well. Experiencing the same behaviour running the latest versions (opentelemetry-collector: 0.85.0, opentelemetry-operator: 0.39.1)

[peter-tar]

Are there any updates regarding this? The only workaround currently is to set up custom grok rules for every single nested field in Datadog which isn't ideal.

[paulius-valiunas]

Are there any updates regarding this? The only workaround currently is to set up custom grok rules for every single nested field in Datadog which isn't ideal.

"isn't ideal" is a huge understatement. A more accurate description would be "as far away as it gets from anything that could be considered a real option"

[mackjmr]

@paulius-valiunas @peter-tar thanks for reporting, this will be fixed in the next release.

If you are using the filelog receiver to collect your json logs, please make sure you are using the json_parser:

    operators:
      - type: json_parser

[peter-tar]

Thanks @mackjmr, I'm looking forward to trying the fix out!