[exporter/datadog] metrics exporter: 413 - Request Entity Too Large

[hanikesn]

Component(s)

exporter/datadog

What happened?

After updating to the latest collector, which uses the native datadog client we're seeing a lot of failures to export metrics with: max elapsed time expired 413 Request Entity Too Large

Collector version

0.69.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

No response

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/datadog: @KSerrania @mx-psi @gbbr @knusbaum @amenasria @dineshg13

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[songy23]

Hi @hanikesn are you batching the metrics? If so can you share the config of your batch processor?

[hanikesn]

Hi @hanikesn are you batching the metrics? If so can you share the config of your batch processor?

We have a batch processor with the following configuration:

  batch:
    timeout: 1s

[songy23]

This probably has to do with the V2 metrics data model change, e.g. in V1 host names are plain strings while in V2 they are in a nested proto message.

Two possible mitigations:

1. Set a smaller threshold for send_batch_size in batchprocessor, this should hopefully reduce the payload size sent to datadogexporter.
2. If the above doesn't work, fall back to using the Zorkian library by adding CLI flag --feature-gates=-exporter.datadogexporter.metricexportnativeclient in your docker file or when starting the collector

[hanikesn]

I configured the batch settings like here:

  batch:
    # Datadog APM Intake limit is 3.2MB. Let's make sure the batches do not
    # go over that.
    send_batch_max_size: 1000
    send_batch_size: 100
    timeout: 10s

And still getting the same error.

[songy23]

From https://docs.datadoghq.com/api/latest/metrics/#submit-metrics:

• In V2 metrics export the payload size limit is 500KB uncompressed, or 5MB after decompression
• In V1 the limit is 3.2MB uncompressed, or 62MB after decompression

#16776 switched metric export to call V2 APIs (which is recommended) instead of V1. While the doc you mentioned still refers to the V1 limit.

Can you try further lowering the send_batch_size for datadog exporter? You might want to have a separate batch processor dedicated for datadog exporter if other exporters expect a larger batch size, e.g.

processors:
  batch:
    timeout: 1s
  batch/2:
    send_batch_max_size: 100
    send_batch_size: 10
    timeout: 10s
...
service:
  pipelines:
    metrics:
      receivers: ...
      processors: [batch/2]
      exporters: [datadog]

[hanikesn]

Can you try further lowering the send_batch_size for datadog exporter?

Thanks @songy23, looks like your suggested configuration works well and we couldn't detect any more failures even during peak hours. Please feel free to close the issue, but I suggest updating the docs here and on Datadogs side as it seems quite likely for more people to hit this issue with the latest version.

[songy23]

Thank you @hanikesn, glad to know it works for you. I'll update the readme in this repo and follow up with updating this page: https://docs.datadoghq.com/opentelemetry/otel_collector_datadog_exporter/?tab=onahost#2-configure-the-datadog-exporter.